Image conversion device and method, and computer-readable recording medium

Abstract

To provide an image conversion device and method capable of converting still images into natural moving images, and a computer-readable recording medium.SOLUTION: An image conversion device that converts images using an artificial neural network includes: an image receiving unit that receives an image from a user; a template acquisition unit that acquires at least one image conversion template; and an image converter that converts a still image into a moving image using the at least one acquired image conversion template. The image converter includes a step of converting the still image received by the image receiving unit into the moving image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to Patent Application No. 10-2019-0141723 filed on November 7, 2019, Patent Application No. 10-2019-0177946 filed on December 30, 2019, Patent Application No. 10-2019-0179927 filed on December 31, 2019, and Patent Application No. 10-2020-0022795 filed on February 25, 2020, all of which are incorporated herein by reference in their entireties. The present invention relates to an image conversion device and method, and a computer-readable recording medium. More specifically, the present invention relates to an image conversion device and method that can convert still images into natural moving images, and a computer-readable recording medium. The present invention relates to a landmark data separation device and method, and a computer-readable recording medium. More particularly, the present invention relates to a landmark data separation device and method, and a computer-readable recording medium, that can more accurately separate landmark data from faces included in images. The present invention relates to a landmark isolation device and method, and a computer-readable recording medium. More specifically, the present invention relates to a landmark isolation device and method, and a computer-readable recording medium, that can isolate landmarks from one frame or a small number of frames. The present invention relates to an image transformation device and method, and a computer-readable recording medium, and more particularly to an image transformation device and method that can generate images that naturally transform according to the characteristics of different images, and a computer-readable recording medium. [Background technology]

[0002] Most mobile personal devices have a built-in camera that can capture static images and moving images such as videos. When a user of a mobile personal device needs a moving image of a desired facial expression, the user must take a picture using the camera built into the mobile personal device. If the moving image does not have the desired facial expression, the user must repeatedly capture the image until a satisfactory result is obtained. Therefore, a method is needed that can replace the desired facial expression with a still image input by the user and convert it into a natural moving image. Active research is being conducted into techniques for analyzing and utilizing human facial images based on facial landmarks obtained by extracting facial key points. Facial landmarks include the results of extracting the origins of key elements on the face, such as the eyes, eyebrows, nose, mouth, and jawline, or extracting contours by connecting these points. Facial landmarks are primarily used in techniques such as facial expression classification, pose analysis, and face synthesis and deformation. However, conventional facial image analysis and utilization techniques based on facial landmarks do not take into account facial appearance features and emotional characteristics when processing facial landmarks, which leads to poor performance. Therefore, there is a need to develop a technique for separating facial landmarks that include emotional characteristics in order to improve the performance of facial image analysis and utilization techniques. Summary of the Invention [Problem to be solved by the invention]

[0003] An object of the present invention is to provide an image conversion device and method, and a computer-readable recording medium, that can convert still images into natural moving images. An object of the present invention is to provide a landmark data separation device and method, and a computer-readable recording medium, that can separate landmark data from a face included in an image more accurately and precisely. An object of the present invention is to provide a landmark separation device and method, and a computer-readable recording medium, that are capable of separating landmarks even from objects with a small amount of data. The present invention aims to provide an image transformation device and method, as well as a computer-readable recording medium, that, when given a target image to be transformed, can use a user's image that is different from the target image to generate an image that matches the user's image but has the characteristics of the target image. [Means for solving the problem]

[0004] An image conversion method according to one embodiment of the present invention is an image conversion method using an artificial neural network, and includes the steps of receiving a static image from a user, obtaining at least one image conversion template, and converting the still image into a moving image using the obtained image conversion template. [Effects of the Invention]

[0005] The present invention may provide an image conversion device and method, as well as a computer-readable recording medium, that provide moving images that have the same effect as moving images captured by a user directly changing their facial expressions, without the need to directly capture the moving images. The present invention may provide an image conversion device and method, as well as a computer-readable recording medium, that provide a user with a moving image generated by converting a still image, thereby providing a fun user experience. The present invention may provide a landmark data separation device and method, as well as a computer-readable recording medium, that can separate landmark data more accurately and precisely from a face included in an image. The present invention may provide a landmark data separation device and method, as well as a computer-readable recording medium, that can separate landmark data that more accurately contains information about facial characteristics and facial expressions included in an image. The present invention may provide a landmark separation device and method, and a computer-readable recording medium, that are capable of separating landmarks even from an object with a small amount of data. The present invention may provide an image transformation device and method, as well as a computer-readable recording medium, that, when given a target image to be transformed, can use a user's image that is different from the target image to generate an image that matches the user's image but has the characteristics of the target image. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram showing a schematic diagram of an environment in which the image conversion method according to the present invention is carried out. [Figure 2] FIG. 2 is a diagram showing a schematic configuration of an image conversion device according to an embodiment of the present invention. [Figure 3] FIG. 3 is a flowchart illustrating an image conversion method according to an embodiment of the present invention. [Figure 4] FIG. 4 is a diagram illustrating an example of an image transformation template according to an embodiment of the present invention. [Figure 5a] FIG. 5a is a diagram illustrating an exemplary process for generating a moving image according to one embodiment of the present invention. [Figure 5b] FIG. 5b is a diagram illustrating an example of a generated moving image according to an embodiment of the present invention. [Figure 6a] FIG. 6a is a diagram illustrating an example of a process for generating a moving image according to another embodiment of the present invention. [Figure 6b] FIG. 6b is a diagram illustrating an example of a moving image generated according to another embodiment of the present invention. [Figure 7] FIG. 7 is a diagram showing the outline of the configuration of an image conversion device according to an embodiment of the present invention. [Figure 8] FIG. 8 is a diagram showing a schematic environment in which the method for extracting landmark data from a face contained in an image according to the present invention is carried out. [Figure 9] FIG. 9 is a diagram schematically illustrating the configuration of a landmark data separation device according to an embodiment of the present invention. [Figure 10] FIG. 10 is a diagram illustrating a method for extracting facial landmark data according to an embodiment of the present invention. [Figure 11] FIG. 11 is a flowchart illustrating a method for extracting various types of landmark data according to one embodiment of the present invention. [Figure 12] FIG. 12 is a diagram illustrating an example process for converting facial expressions contained in an image according to another embodiment of the present invention. [Figure 13] FIG. 13 is a comparison table illustrating the effect of converting facial expressions contained in an image using the landmark data separation method according to the present invention. [Figure 14] FIG. 14 is a diagram schematically illustrating the configuration of a landmark data separation device according to an embodiment of the present invention. [Figure 15] FIG. 15 is a diagram schematically illustrating an environment in which the landmark separating device according to the present invention operates. [Figure 16] FIG. 16 is a flowchart that schematically illustrates a landmark separation method according to one embodiment of the present invention. [Figure 17] FIG. 17 is a diagram illustrating a method for calculating a transformation matrix according to an embodiment of the present invention. [Figure 18] FIG. 18 is a diagram schematically illustrating the configuration of a landmark separating device according to one embodiment of the present invention. [Figure 19] FIG. 19 is a diagram illustrating an exemplary method for recreating a face using the present invention. [Figure 20] FIG. 20 is a diagram illustrating an environment in which the image transformation device and image transformation method according to the present invention operate. [Figure 21] FIG. 21 is a flowchart illustrating an image deformation method according to an embodiment of the present invention. [Figure 22] FIG. 22 is a diagram illustrating an example of a result of performing an image transformation method according to an embodiment of the present invention. [Figure 23] FIG. 23 is a diagram showing a schematic configuration of an image transformation device according to an embodiment of the present invention. [Figure 24] FIG. 24 is a diagram schematically illustrating the configuration of a landmark acquisition unit according to an embodiment of the present invention. [Figure 25] FIG. 25 is a diagram schematically illustrating a configuration of a second encoder according to an embodiment of the present invention. [Figure 26] FIG. 26 is a diagram showing a schematic diagram of a blender structure according to one embodiment of the present invention. [Figure 27] FIG. 27 is a diagram illustrating the structure of a decoder according to one embodiment of the present invention. [Figure 28a-c] Figures 28a-c show examples of identity preservation failure and the improvement produced by the proposed method. Figure 28a shows the interference of the driver shape, Figure 28b shows the loss of target identity details, and Figure 28c shows the warping failure for large poses. [Figure 29] Figure 29 shows the overall structure of MarioNETte. [Figure 30] Figure 30 shows the structure of an image attention block. [Figure 31] FIG. 31 shows the structure of the target feature alignment. [Figure 32] FIG. 32 shows the structure of the landmark decomposition section. [Figure 33] Figure 33 shows images generated by the proposed method, the baseline, and replaying different identities on CelebV under a single imaging setting. [Figure 34] Figure 34 shows the evaluation results of VoxCeleb1 in a self-repeat setting. [Figure 35] Figure 35 shows the evaluation results of replaying different identities on CelebV. [Figure 36]Figure 36 shows the results of a user study where different identities were reprised on CelebV. [Figure 37] Figure 37 shows a comparison of ablation models for recapitulating different identities in CelebV. [Figure 38] Figure 38a shows the driver and target images superimposed with the attention map, and Figure 38b shows the failed case of +Alignment and the improved result produced by MarioNETte. [Figure 39] FIG. 39 shows an example of rasterized facial landmarks. [Figure 40] Figure 40 shows a comparison of ablation models for the self-replica setting on the VoxCeleb1 dataset. [Figure 41] Figure 41 shows the inference speed of each component of the model. [Figure 42] Figure 42 shows the inference speed of the overall model for generating a single image from K target images. [Figure 43] Figure 43 shows qualitative results of the single-image-replay ablation model under different identity settings in CelebV. [Figure 44] Figure 44 shows the qualitative results of the ablation model, replicated several times under different identity settings in CelebV. [Figure 45] Figure 45 shows qualitative results of the single-image self-replay setting on VoxCeleb1. [Figure 46] Figure 46 shows qualitative results of the multiple self-replay setup on VoxCeleb1. [Figure 47] Figure 47 shows the qualitative results of single-image replays under different identity settings in VoxCeleb1. [Figure 48] Figure 48 shows the qualitative results of several re-imaging runs under different identity settings in VoxCeleb1. [Figure 49] Figure 49 shows qualitative results of the single-image self-replay setting on CelebV. [Figure 50]Figure 50 shows qualitative results of the multiple imaging self-replay setup on CelebV. [Figure 51] Figure 51 shows the qualitative results of several re-imaging runs under different identity settings in CelebV. [Figure 52] Figure 52 shows an example of a failure produced by MarioNETte+LTd during a single replay under different identity settings in VoxCeleb1. DETAILED DESCRIPTION OF THE INVENTION

[0007] The advantages and features of the present invention, as well as methods for achieving them, will become apparent from the following detailed description of the embodiments taken in conjunction with the accompanying drawings. In this regard, the embodiments of the present invention may have various forms and are not limited to the description set forth herein. Rather, these embodiments will comprehensively understand the present disclosure and fully convey the scope of the present disclosure to those skilled in the art, and the present disclosure is defined solely by the appended claims. Like reference numerals refer to like elements throughout the specification. Although terms such as "first" and "second" are used to describe various components, these components are not limited to the above terms. The above terms may be used to distinguish one component from another. Therefore, the first component described below may be the second component within the technical concept of the present invention. The terms used in this specification are for the purpose of describing the embodiments and are not intended to limit the present invention. In this specification, the singular form includes the plural form unless otherwise specified in the text. The words "comprises" or "comprising" used in this specification do not exclude the presence or addition of one or more other components or steps in the component or step mentioned. Unless otherwise defined, all terms used herein may be interpreted in a manner that would be commonly understood by a person of ordinary skill in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless they are clearly and specifically defined.

[0008] Fig. 1 is a diagram schematically illustrating an environment in which an image conversion method according to the present invention is executed. Referring to Fig. 1, the environment in which an image conversion method according to the present invention is executed may include a server 10 and a terminal 20 connected to the server 10. For convenience of explanation, only one terminal is shown in Fig. 1, but multiple terminals may be included. The explanation regarding the terminal 20 may also be applied to additional terminals, unless otherwise noted. In an embodiment of the present invention, the server 10 may receive an image from the terminal 20, convert the received image into an arbitrary format, and then transmit the converted image to the terminal 20. Alternatively, the server 10 may function as a platform that provides a service that the terminal 20 may connect to and use. The terminal 20 may convert an image selected by a user of the terminal 20 and transmit the converted image to the server 10. The server 10 may be connected to a communication network. The server 10 may be connected to other external devices via the communication network. The server 10 may transmit data to the other devices connected to the server 10, or may receive data from the other devices. The communication network connected to the server 10 may include a wired communication network, a wireless communication network, or a combined communication network. The communication network may include a mobile communication network such as 3G, LTE, or LTE-A. The communication network may include a wired or wireless communication network such as Wi-Fi, UMTS / GPRS, or Ethernet. The communication network may include a local area network such as Magnetic Secure Transmission (MST), RFID (Radio Frequency Identification), NFC (Near Field Communication), Zigbee, Z-Wave, Bluetooth, Bluetooth Low Energy (BLE), or InfraRed communication (IR). The communication network may include a local area network (LAN), a metropolitan area network (MAN), or a wide area network (WAN). The server 10 may be connected to the terminal 20 via a communication network. When the server 10 is connected to the terminal 20, the server 10 may transmit and receive data to and from the terminal 20 via the communication network. The server 10 may execute any calculation using data received from the terminal 20. The server 10 may transmit the calculation result to the terminal 20. The terminal 20 may be a desktop computer, a laptop computer, a smartphone, a smart tablet, a smart watch, a mobile terminal, a digital camera, a wearable device, or a portable electronic device, etc. The terminal 20 may execute a program or an application.

[0009] FIG. 2 is a diagram showing a schematic configuration of an image conversion device according to an embodiment of the present invention. 2, an image conversion device 100 according to an embodiment of the present invention includes an image receiving unit 110, a template acquisition unit 120, and an image conversion unit 130. The image conversion device 100 may be configured by the server 10 or the terminal 20 described with reference to FIG. 1. Therefore, each component included in the image conversion device 100 may also be configured by the server 10 or the terminal 20. The image receiving unit 110 receives an image from a user. The image may include the user's face and may be a still image or a static image. Meanwhile, the size of the user's face included in the image may vary from image to image. For example, the size of the face included in image 1 may be 100x100 pixels, and the size of the face included in image 2 may be 200x200 pixels.

[0010] The image receiving unit 110 may extract only the face area from the image received from the user and then provide it to the image converting unit 130 . The image receiving unit 110 may extract a region corresponding to the user's face from the image containing the user's face, with the region having a predetermined size. For example, if the predetermined size is 100×100 and the region corresponding to the user's face included in the image has a size of 200×200, the image receiving unit 110 may reduce the 200×200 image to 100×100 and then extract the image. Alternatively, the 200×200 image may be extracted and then converted into a 100×100 image. The template acquisition unit 120 acquires at least one image transformation template. The image transformation template may be understood as a tool capable of transforming an image received by the image receiving unit 110 into a new image of a specific form. For example, when an image received by the image receiving unit 110 includes a neutral face of a user, a specific image transformation template can be used to generate a new image including a smiling face of the user. The image transformation template may be predetermined to any template or may be selected by the user. The image conversion unit 130 may receive a still image corresponding to the face region from the image receiving unit 110. Furthermore, the image conversion unit 130 may convert the still image into a moving image using an image conversion template acquired by the template acquisition unit 120.

[0011] FIG. 3 is a flowchart illustrating an image conversion method according to an embodiment of the present invention. Referring to FIG. 3, an image transformation method according to an embodiment of the present invention may include step S110 of receiving a still image, step S120 of obtaining an image transformation template, and step S130 of generating a moving image. The image conversion method according to the present invention is an image conversion method using an artificial neural network, and can acquire a still image in step S110. The still image may include a user's face or may include one frame. In step S120, at least one image transformation template may be obtained from the plurality of image transformation templates stored in the image transformation device 100. The image transformation template may be selected by the user from the plurality of image transformation templates stored in the image transformation device 100. The image transformation template may be understood as a tool that can transform the image received in step S110 into a new image of a specific form. For example, if the image received in step S110 contains a neutral face of a user, using a specific image transformation template may generate a new image containing a smiling face of the user. In another embodiment, when the image received in step S110 contains a smiling face of the user, another specific image transformation template may be used to generate a new image containing an angry face of the user. In some embodiments, at least one reference image may be received from the user in step S120. For example, the reference image may be an image of the user or an image of another person selected by the user. If the user selects a reference image instead of selecting one of a plurality of defined templates, the reference image may be acquired as the image transformation template. That is, the reference image may be understood to function similarly to the image transformation template. In step S130, the still image may be converted into a moving image using the acquired image conversion template. To convert the still image into a moving image, texture information may be extracted from the user's face included in the still image. The texture information may be color and visual texture information of the user's face.

[0012] Furthermore, to convert a still image into a moving image, landmark information may be extracted from an area corresponding to a person's face included in the image conversion template. Feature point information may be obtained from a specific shape, pattern, color, or combination thereof included in the person's face based on an image processing algorithm. The image processing algorithm may be, but is not limited to, SIFT (Scale Invariant Feature Transform), HOG (HiStogram of Oriented Gradient), Haar feature, Ferns, LBP (Local Binary Pattern), or MCT (Modified Census Transform). The video may be generated by combining the texture information and landmark information. In some embodiments, the video may include a plurality of frames. The video may have a first frame corresponding to the still image and a last frame corresponding to the image transformation template. For example, the facial expression of the user included in the still image may be the same as the face included in the first frame of the video. In addition, by combining the texture information and landmark information, the facial expression of the user included in the still image may be transformed according to the landmark information, and the last frame of the video may include a frame corresponding to the transformed user face. When generating a video using an artificial neural network, the video may gradually change from the user's facial expression included in the still image to the user's facial expression converted according to the landmark information. That is, at least one or more frames may be included between the first and last frames of the video, and the facial expression included in each of the at least one or more frames may gradually change. In this way, by using an artificial neural network, it is possible to generate moving images that have the same effect as moving images captured by a user directly changing their facial expressions, without having to capture the moving images themselves.

[0013] FIG. 4 is a diagram illustrating an example of an image transformation template according to an embodiment of the present invention. The image conversion device 100 may store a plurality of image conversion templates. Each of the plurality of image conversion templates may include outline images corresponding to eyebrows, eyes, and a mouth. The plurality of image conversion templates may correspond to various facial expressions, such as a sad expression, a happy expression, a winking expression, a gloomy expression, a neutral expression, a surprised expression, and an angry expression, and each of the plurality of image conversion templates may include information on a different facial expression. The outline images corresponding to the various facial expressions are different from one another. Therefore, each of the plurality of image conversion templates may include a different outline image from one another. Referring to FIG. 2, the image conversion unit 130 may extract landmark information from an outline image included in the image conversion template.

[0014] FIG. 5a is a diagram illustrating an exemplary process for generating a moving image according to one embodiment of the present invention. 4 and 5a, there are shown a still image 31, an image transformation template 32, and a moving image 33 generated using the still image 31 and the image transformation template 32. For example, the still image 31 may include a smiling face of a user. The image transformation template 32 may include an outline image corresponding to the eyebrows, eyes, and mouth of a winking, smiling face. On the other hand, although the moving image 33 shown in FIG. 5a is considered to include only one frame, the moving image 33 may also be understood to indicate the last frame that constitutes the moving image generated by the image conversion unit 130 or step S130. The image conversion device 100 may extract texture information of an area corresponding to the user's face from the still image 31. The image conversion device 100 may also extract landmark information from the image conversion template 32. The image conversion device 100 may generate a moving image 33 by combining the texture information of the still image 31 and the landmark information of the image conversion template 32. The motion picture 33 is shown as a single image containing the user's winking face, but the motion picture 33 includes multiple frames. A motion picture 33 including multiple frames will be described with reference to Figure 5b. FIG. 5b is a diagram illustrating an example of a generated moving image according to an embodiment of the present invention. 5a and 5b, there may be at least one frame between the first frame 33_1 and the last frame 33_n of the video 33. For example, the still image 31 may correspond to the first frame 33_1 of the video 33. Also, the image including the user's winking face may correspond to the last frame 33_n of the video 33. At least one frame between the first frame 33_1 and the last frame 33_n of the moving image 33 may each include an image of the user's face with the eyes gradually closing.

[0015] FIG. 6a is a diagram illustrating an example of a process for generating a moving image according to another embodiment of the present invention. 4 and 6a, there are shown a still image 41, a reference image 42, and a moving image 43 generated using the still image 41 and the reference image 42. For example, the still image 41 may include a smiling face of a user. The reference image 42 may include a winking, smiling face. The face included in the reference image 42 may be the face of a person different from the user. On the other hand, although the moving image 43 shown in FIG. 6a is considered to include only one frame, the moving image 43 may also be understood to indicate the last frame that constitutes the moving image generated by the image conversion unit 130 or step S130. The image conversion device 100 may extract texture information of an area corresponding to the user's face from the still image 41. The image conversion device 100 may also extract landmark information from the reference image 42. The image conversion device 100 may extract landmark information in areas corresponding to the eyebrows, eyes, and mouth of the face included in the reference image 42. The image conversion device 100 may generate a moving image 43 by combining the texture information of the still image 41 and the landmark information of the reference image 42. The motion picture 43 is shown as a single image containing the user's smiling and winking face, but the motion picture 43 includes multiple frames. A motion picture 43 including multiple frames will be described with reference to Figure 6b.

[0016] FIG. 6b is a diagram illustrating an example of a moving image generated according to another embodiment of the present invention. 6a and 6b, there may be at least one frame between the first frame 43_1 and the last frame 43_n of the video 43. For example, the still image 41 may correspond to the first frame 43_1 of the video 43. Alternatively, the image including the user's smiling and winking face may correspond to the last frame 43_n of the video 43. At least one frame between the first frame 43_1 and the last frame 43_n of the moving image 43 may each include an image of the user's face with the eyes gradually closing and the mouth gradually opening.

[0017] FIG. 7 is a diagram showing the outline of the configuration of an image conversion device according to an embodiment of the present invention. 7, the image conversion device 200 may include a processor 210 and a memory 220. Those skilled in the art will understand that in addition to the components shown in FIG. The image conversion device 200 may be similar to or identical to the image conversion device 100 shown in Fig. 2. The image receiving unit 110, the template acquisition unit 120, and the image conversion unit 130 included in the image conversion device 100 may be further included in a processor 210. The processor 210 controls all operations of the image conversion device 200 and may include at least one processor such as a CPU. The processor 210 may include at least one specialized processor corresponding to each function, or may be a single integrated processor. Memory 220 may store programs, data, or files related to the artificial neural network. Memory 220 may store instructions executable by processor 210. Processor 210 may execute programs stored in memory 220, read data or files stored in memory 220, or store new data. Memory 220 may also store program instructions, data files, data structures, etc., alone or in combination.

[0018] The processor 210 may obtain a still image from the input image. The still image may include the user's face and may include a single frame. The processor 210 may read at least one image transformation template from among a plurality of image transformation templates stored in the memory 220. Alternatively, the processor 210 may read at least one reference image stored in the memory 220. For example, the at least one reference image may be input by a user. The reference image may be an image of the user or an image of another person selected by the user. If the user does not select one of the defined templates but selects a reference image, the reference image may be acquired as the image transformation template. The processor 210 may convert the still image into a moving image using the acquired image conversion template. To convert the still image into a moving image, the processor 210 may extract texture information from the user's face included in the still image. The texture information may be color and visual texture information of the user's face. Furthermore, to convert a still image into a moving image, landmark information may be extracted from an area corresponding to a person's face included in the image conversion template. Feature point information may be obtained from a specific shape, pattern, color, or combination thereof included in the person's face based on an image processing algorithm. The image processing algorithm may be, but is not limited to, SIFT (Scale Invariant Feature Transform), HOG (Histogram of Oriented Gradient), Haar feature, Ferns, LBP (Local Binary Pattern), or MCT (Modified Census Transform).

[0019] The moving image may be generated by combining the texture information and landmark information. The moving image may include a plurality of frames. The moving image may have a frame corresponding to the still image as a first frame and a frame corresponding to the image transformation template as a last frame. For example, the facial expression of the user included in the still image may be the same as the face included in the first frame of the video. Furthermore, by combining the texture information and landmark information, the facial expression of the user included in the still image may be transformed according to the landmark information, and the last frame of the video may include a frame corresponding to the transformed user face. The video generated by processor 210 may have a shape as shown in Figures 5b and 6b. The processor 210 may store the generated motion images in the memory 220 and output the motion images so that the user can view them. 1 to 7, when a user uploads a still image to the user's terminal 20, the image conversion device 200 may convert the still image into a moving image and provide it to the user. Even if the user does not directly capture the moving image, the user can be provided with a moving image that has the same effect as a moving image captured by directly changing facial expressions. Furthermore, the image conversion device 200 can provide a user with a fun user experience by providing the user with a moving image generated by converting the moving image into a still image.

[0020] FIG. 8 is a diagram showing a schematic environment in which the method for extracting landmark data from a face contained in an image according to the present invention is carried out. Referring to Fig. 8, an environment in which the landmark data extraction method according to the present invention is executed may include a server 10-1 and a terminal 20-1 connected to the server 10-1. For convenience of explanation, only one terminal is shown in Fig. 8, but multiple terminals may be included. The explanation regarding the terminal 20-1 may also be applied to additional terminals, unless otherwise noted. In an embodiment of the present invention, server 10-1 may receive an image from terminal 20-1, extract landmark data from a face contained in the received image, calculate necessary data from the extracted landmark data, and then transmit the calculated data to terminal 20-1. Alternatively, server 10-1 may function as a platform that provides services that terminal 20-1 may connect to and use. Terminal 20-1 may extract landmark data from a face included in an image, calculate necessary data from the extracted landmark data, and then transmit the calculated data to server 10-1.

[0021] The server 10-1 may be connected to a communication network. The server 10-1 may be connected to other external devices via the communication network. The server 10-1 may transmit data to the other devices connected to the server 10-1, and may receive data from the other devices. The communication network connected to the server 10-1 may include a wired communication network, a wireless communication network, or a combined communication network. The communication network may include a mobile communication network such as 3G, LTE, or LTE-A. The communication network may include a wired or wireless communication network such as Wi-Fi, UMTS / GPRS, or Ethernet. The communication network may include a local area network such as Magnetic Secure Transmission (MST), RFID (Radio Frequency Identification), NFC (Near Field Communication), Zigbee, Z-Wave, Bluetooth, Bluetooth Low Energy (BLE), or InfraRed communication (IR). The communication network may include a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), or the like. The server 10-1 may be connected to the terminal 20-1 via a communication network. When the server 10-1 and the terminal 20-1 are connected to each other, the server 10-1 may transmit and receive data to and from the terminal 20-1 via the communication network. The server 10-1 may execute any calculation using data received from the terminal 20-1. The server 10-1 may transmit the calculation result to the terminal 20-1. The terminal 20-1 may be a desktop computer, a laptop computer, a smartphone, a smart tablet, a smart watch, a mobile terminal, a digital camera, a wearable device, or a portable electronic device, etc. The terminal 20-1 may execute a program or an application.

[0022] FIG. 9 is a diagram schematically illustrating the configuration of a landmark data separation device according to an embodiment of the present invention. 9, a landmark data separation device 100-1 according to an embodiment of the present invention may include an image receiving unit 110-1, a landmark data calculation unit 120-1, and a landmark data storage unit 130-1. The landmark data separation device 100-1 may be configured by the server 10-1 or the terminal 20-1 described with reference to Fig. 8. Therefore, each of the components included in the landmark data separation device 100-1 may also be configured by the server 10-1 or the terminal 20-1. Image receiving unit 110-1 may receive multiple images from a user. Each of the multiple images may include only one person. That is, each of the multiple images may include the face of one person, and the people included in the multiple images may be different people. The image receiving unit 110-1 may extract only the face area from each of the multiple images, and then provide the extracted face area to the landmark data calculation unit 120-1. The landmark data calculation unit 120-1 may calculate face landmark data included in each of the multiple images, average landmark data of all faces included in the multiple images, characteristic landmark data of a specific face included in a specific image among the multiple images, and expression landmark data of a specific face. In some embodiments, the landmark data may be the result of extracting face key points. A method for extracting landmark data is described with reference to FIG.

[0023] FIG. 10 is a diagram illustrating a method for extracting facial landmark data according to an embodiment of the present invention. The landmark data may be obtained by extracting the starting points of major facial features such as the eyes, eyebrows, nose, mouth, and jawline, or by extracting contours drawn by connecting these points. The landmark data may be used in techniques such as facial expression classification, pose analysis, facial synthesis of different people, or facial deformation. 9 again, landmark data calculation unit 120-1 may calculate average landmark data of faces included in multiple images. The average landmark data may be the result of extracting an average human face shape. The landmark data calculation unit 120-1 may calculate landmark data from a specific image that includes a specific face among multiple images. More specifically, it may calculate specific face landmark data included in a specific frame among multiple frames included in the specific image. Furthermore, the landmark data calculation unit 120-1 may calculate characteristic landmark data of a specific face included in a specific image among multiple images. The characteristic landmark data may be calculated based on the facial landmark data included in each of multiple frames included in the specific image. The data calculation unit 120-1 may also calculate average landmark data, landmark data for a specific frame, and characteristic landmark data to calculate facial expression landmark data for a specific frame in a specific image. For example, the facial expression landmark data may correspond to specific facial expressions or movement information of major elements such as the eyes, eyebrows, nose, mouth, and jawline. The landmark data storage unit 130-1 may store data calculated by the landmark data calculation unit 120-1. For example, the landmark data storage unit 130-1 may store average landmark data, landmark data for a specific frame, characteristic landmark data, and facial landmark data, which are calculated by the landmark data calculation unit 120-1.

[0024] FIG. 11 is a flowchart illustrating a method for extracting various types of landmark data according to one embodiment of the present invention. 9 and 11, in step S1100, the landmark data separation device 100-1 may receive a plurality of images. Each of the plurality of images may include only one person. That is, each of the plurality of images may include the face of one person, and the people included in the plurality of images may be different people. In step S1200, the landmark data separating device 100-1 separates the average landmark data I m The average landmark data I m can be expressed as follows:

number

[0025] For example, landmark data for a specific frame I (c、t) may be the main origin information of a specific face included in the t-th frame of the c-th image among the multiple images C. That is, the specific image may be the c-th image, and the specific frame may be the t-th frame. In step S1400, the landmark data separating device 100-1 separates the characteristic landmark data I of a specific face included in the c-th image. id(c) Characteristic landmark data I id(c) can be expressed as follows:

number

[0026] More specifically, the landmark data separation device 100-1 separates specific facial expression landmark data I included in the t-th frame of the c-th image. exp(c、t) The facial landmark data I exp(c、t) can be expressed as follows:

number

[0027] FIG. 12 is a diagram illustrating an example process for converting facial expressions contained in an image according to another embodiment of the present invention. 11 and 12, the server 10-1 or the terminal 20-1 receives the facial landmark data I separated from the landmark data separation device 100-1. exp(c、t) , average landmark data I m , and characteristic landmark data I id(c) may be utilized to convert only the facial expression into the facial expression of the face included in the second image 400 while maintaining the appearance of the face included in the first image 300. For example, the first image 300 is c x Among the frames contained in the image, x The second image 400 may correspond to the cth frame among the plurality of images. y Among the frames contained in the image, y It may correspond to the cth frame. x The second image and c y The first and second images may be different images. The facial landmark data included in the first image 300 may be separated as follows.

number

[0028] The facial landmark data included in the second image 400 may be separated as follows.

number

number

[0029] FIG. 13 is a comparison table illustrating the effect of converting facial expressions contained in an image using the landmark data separation method according to the present invention. The MarioNETte model converts facial expressions contained in images without using landmark data separation methods. When using the MarioNETte model, the naturalness of the converted image was measured to be 0.147. The MarioNETte+LT model converts facial expressions contained in an image using a landmark data separation method. When using the MarioNETte model, the naturalness of the converted image was measured to be 0.280. In other words, images converted using the MarioNETte+LT model were confirmed to be 1.9 times more natural than images converted using the MarioNETte model.

[0030] FIG. 14 is a diagram schematically illustrating the configuration of a landmark data separation device according to an embodiment of the present invention. 14, the landmark data separation device 200-1 may include a processor 210-1 and a memory 220-1. A person skilled in the art will understand that in addition to the components shown in FIG. 14, other general components may also be included. The image conversion device 200-1 may be similar to or identical to the landmark data separation device 100-1 shown in Fig. 9. The image receiving unit 110-1 and the landmark data calculation unit 120-1 included in the landmark data separation device 100-1 may be included in the processor 210-1. The processor 210-1 controls the overall operation of the landmark data separation device 200-1 and may include at least one processor such as a CPU. The processor 210-1 may include at least one specialized processor corresponding to each function, or may be a single integrated processor. The memory 220-1 may store programs, data, or files that control the landmark data separation device 200-1. The memory 220-1 may store instructions that can be executed by the processor 210-1. The processor 210-1 may execute programs stored in the memory 220-1, read data or files stored in the memory 220-1, or store new data. The memory 220-1 may also store program instructions, data files, data structures, etc., alone or in combination.

[0031] Processor 210-1 may receive multiple images, each of which may include only one person, i.e., each of the multiple images may include the face of one person, and the people included in the multiple images may be different people. The processor 210-1 may store the received images in the memory 220-1. The processor 210-1 calculates landmark data I of each face included in a plurality of images C. (c、t) The landmark data separation device 100-1 may calculate the average value of all the extracted landmark data. The calculated average value is used as the average landmark data I m It may correspond to. The processor 210-1 performs an I / F search for landmark data for a specific frame among a plurality of frames of a specific image that includes a specific face among a plurality of images. (c、t) may be calculated. Landmark data for a specific frame I (c、t) may be the main origin information of a specific face included in the t-th frame of the c-th image among the multiple images C. That is, the specific image may be the c-th image, and the specific frame may be the t-th frame. The processor 210-1 selects characteristic landmark data I of a specific face included in the c-th image. id(c)The c-th image contains multiple frames that include various facial expressions of a specific face. Therefore, the characteristic landmark data I id(c) To calculate this, the processor 210-1 uses specific facial expression landmark data I included in the c-th image. exp Average value of JPEG2025023956000021.jpg1541 can be set to 0. Therefore, characteristic landmark data I id(c) is a specific facial expression landmark data I exp Average value of The calculation may be performed without taking JPEG2025023956000022.jpg1541 into consideration. Characteristic Landmark Data I id(c) calculates landmark data for each of the multiple frames included in the c-th image, and calculates the average landmark data of the landmark data for each of the multiple frames. JPEG2025023956000023.jpg1535 is calculated, and the average landmark data of the calculated cth image is calculated. Average landmark data from multiple images from JPEG2025023956000024.jpg1535 I m may be defined as the value minus

[0032] The processor 210-1 calculates specific facial expression landmark data I included in the t-th frame of the c-th image. exp(c、t) The facial landmark data I exp(c、t) may correspond to a specific facial expression included in the t-th frame and movement information of the eyes, eyebrows, nose, mouth, jaw line, etc. included in the specific face. More specifically, the facial landmark data I exp(c、t) is the landmark data for a specific frame. (c、t) Average landmark data from I m and characteristic landmark data I id(c) may be defined as the value minus The processor 210-1 receives the separated facial landmark data I exp(c、t) , average landmark data Im , and characteristic landmark data I id(c) may be stored in memory 220-1. As will be described with reference to FIGS. 8 to 14, the landmark data separation devices 100-1 and 200-1 according to an embodiment of the present invention can separate more accurate and precise landmark data from a face included in an image. Furthermore, the landmark data separation devices 100-1 and 200-1 can separate landmark data that more accurately includes information about the characteristics and expressions of the faces contained in the images. Furthermore, the server 10-1 or the terminal 20-1 including the landmark data separation devices 100-1 and 200-1 receives the separated facial landmark data I exp(c、t) , average landmark data I m , and characteristic landmark data I id(c) By utilizing this, it is possible to realize a technology that naturally converts the facial expression of a face contained in a first image into the facial expression of a face contained in a second image while maintaining the appearance of the face contained in the first image.

[0033] Fig. 15 is a diagram schematically illustrating an environment in which a landmark separating device according to the present invention operates. Referring to Fig. 15, the environment in which a first terminal 2000 and a second terminal 3000 operate may include a server 1000, and the first terminal 2000 and the second terminal 3000 connected to the server 1000. For convenience of explanation, Fig. 15 shows only two terminals, i.e., the first terminal 2000 and the second terminal 3000, but two or more terminals may be included. The explanation regarding the first terminal 2000 and the second terminal 3000 may apply to additional terminals unless otherwise noted. The server 1000 may be connected to a communication network. The server 1000 may be connected to other external devices via the communication network. The server 1000 may transmit data to the other devices connected to the server 1000, or may receive data from the other devices. The communication network connected to the server 1000 may include a wired communication network, a wireless communication network, or a combined communication network. The communication network may include a mobile communication network such as 3G, LTE, or LTE-A. The communication network may include a wired or wireless communication network such as Wi-Fi, UMTS / GPRS, or Ethernet. The communication network may include a local area network such as Magnetic Secure Transmission (MST), Radio Frequency Identification (RFID), Near Field Communication (NFC), ZigBee, Z-Wave, Bluetooth, Bluetooth Low Energy (BLE), or Infrared communication (IR). The communication network may include a local area network (LAN), a metropolitan area network (MAN), or a wide area network (WAN).

[0034] The server 1000 may receive data from at least one of the first terminal 2000 and the second terminal 3000. The server 1000 may perform a calculation using the data received from at least one of the first terminal 2000 and the second terminal 3000. The server 1000 may transmit the result of the calculation to at least one of the first terminal 2000 and the second terminal 3000. The server 1000 may receive an intermediation request from at least one of the first terminal 2000 and the second terminal 3000. The server 1000 may select a terminal to which the intermediation request is to be transmitted. For example, the server 1000 may select the first terminal 2000 and the second terminal 3000. The server 1000 may mediate a communication connection between the selected first terminal 2000 and second terminal 3000. For example, the server 1000 may mediate a video call connection between the first terminal 2000 and the second terminal 3000, or may mediate a text transmission / reception connection. The server 1000 may transmit connection information related to the first terminal 2000 to the second terminal 3000, or may transmit connection information related to the second terminal 3000 to the first terminal 2000. The connection information regarding the first terminal 2000 may include, for example, the IP address and port number of the first terminal 2000. Upon receiving the connection information regarding the second terminal 3000, the first terminal 2000 may attempt to connect to the second terminal 3000 using the received connection information.

[0035] A video call session between the first terminal 2000 and the second terminal 3000 may be established by a successful attempt to connect the first terminal 2000 to the second terminal 3000 or a successful attempt to connect the second terminal 3000 to the first terminal 2000. Through the video call session, the first terminal 2000 may transmit images and sounds to the second terminal 3000. The first terminal 2000 may encode the images and sounds into digital signals and transmit the encoded results to the second terminal 3000. The first terminal 2000 may receive images and sounds encoded in a digital signal and decode the received images and sounds. Through the video call session, the second terminal 3000 may transmit images and sounds to the first terminal 2000. Also, through the video call session, the second terminal 3000 may receive images and sounds from the first terminal 2000. In this way, the user of the first terminal 2000 and the user of the second terminal 3000 can make a video call with each other. The first terminal 2000 and the second terminal 3000 may be, for example, a desktop computer, a laptop computer, a smartphone, a smart tablet, a smart watch, a mobile terminal, a digital camera, a wearable device, or a portable electronic device. The first terminal 2000 and the second terminal 3000 may execute a program or an application. The first terminal 2000 and the second terminal 3000 may be the same type of device or may be different types of devices.

[0036] 16 is a flowchart illustrating a landmark separation method according to an embodiment of the present invention. Referring to FIG. 16, the landmark separation method according to an embodiment of the present invention includes a step of receiving a face image and landmark information (S210), a step of estimating a transformation matrix (S220), a step of calculating expression landmarks (S230), and a step of calculating identity landmarks (S240). In step S210, a facial image of a first person and landmark information corresponding to the facial image are received. Here, the landmarks may be understood as facial landmarks of the facial image. The landmarks may refer to major elements of a face, such as eyes, eyebrows, nose, mouth, jawline, etc. The landmark information may also include information about the position, size, or shape of the major elements of the face, and may also include information about the color or texture of the major elements of the face.

[0037] The first person refers to any person, and in step S210, a face image of the person and landmark information corresponding to the face image are received. The landmark information is obtained using known technology, and any known method may be used. The method of acquiring the landmarks does not limit the present invention. In step S220, a transformation matrix corresponding to the landmark information is estimated. The transformation matrix may constitute the landmark information together with a predetermined unit vector. For example, the first landmark information may be calculated by multiplying the unit vector by the first transformation matrix. Also, the second landmark information may be calculated by multiplying the unit vector by the second transformation matrix. The transformation matrix converts high-dimensional landmark information into low-dimensional data and may be used in Principal Component Analysis (PCA). PCA is a dimension reduction method that converts variables in a high-dimensional space into variables in a low-dimensional space by searching for new mutually orthogonal axes while maximally preserving the variance of the data. PCA first finds a hyperplane that is closest to the data, and then projects the data onto the low-dimensional hyperplane to reduce the dimension of the data. In PCA, the unit vector defining the i-th axis is taken as the i-th principal component (PC), and high-dimensional data can be converted to low-dimensional data by linearly combining these axes.

number

[0038] Therefore, in step S220, the facial image of the first person and the landmark information corresponding to the facial image are received as input, and a transformation matrix is ​​estimated and output therefrom. On the other hand, the learning model may be trained to classify landmark information into multiple semantic groups corresponding to the right eye, left eye, nose, and mouth, and to output PCA transform coefficients corresponding to each of the multiple semantic groups. In this case, the semantic groups are not necessarily classified to correspond to the right eye, left eye, nose, and mouth, but may be classified to correspond to the eyebrows, eyes, nose, mouth, and jaw line, or may be classified to correspond to the eyebrows, right eye, left eye, nose, mouth, jaw line, ears, etc. In step S120, the landmark information may be classified into semantic groups in smaller units according to the learning model, and PCA transformation coefficients corresponding to the classified semantic groups may be estimated. In step S230, the expression landmarks of the first person are calculated using the transformation matrix. Landmark information can be decomposed into a plurality of sub-landmark information, but in the present invention, the landmark information is represented as follows:

number

[0039] Here, l(c, t) is the landmark information of the t-th frame of the video containing person c, and l m is the mean facial landmark information for a person, l id (c) is the facial landmark of identity geometry information of person c, exp (c, t) denotes the facial landmark of expression geometry of person c in the t-th frame of a video including person c. In other words, the landmark information for a specific person in a specific frame may be represented as the sum of the average landmark information for all people's faces, the identity landmark information for the specific person only, and the facial expression and movement information for the specific person in the specific frame. The average landmark information can be defined by the following formula, and may be calculated based on many images that can be collected in advance.

number

number

[0040] In other words, b exp denotes the eigenvectors discussed previously, and high-dimensional representation landmarks may be defined by combinations of low-dimensional eigenvectors. expmeans the total number of expressions and movements that person c can make using the right eye, left eye, nose, mouth, etc. Therefore, the expression landmarks of the first person may be defined as a set of expression information for each of the main parts of the face, that is, the right eye, the left eye, the nose, and the mouth. k (c, t) may exist corresponding to each eigenvector. The learning model may be trained to estimate the PCA coefficient α(c, t) using a photo x(c, t) of a person c whose landmark information is to be separated and landmark information l(c, t) as input, as shown in Equation 8. Through such training, the learning model may estimate a PCA coefficient from an image of a specific person and its corresponding landmark information, or may estimate the low-dimensional eigenvector. When applying a trained neural network, the photo x(c', t) of the person c' for which landmark separation is to be performed and the landmark information l(c', t) are input to the neural network, and the PCA transformation matrix is ​​estimated. exp are the predicted (estimated) PCA coefficients and b exp , we may use it to estimate representation landmarks as follows:

number

[0041] In step S240, the representation landmarks are used to calculate the identity landmarks of the first person. As described with reference to Equation 2, landmark information may be defined as the sum of average landmark information, identity landmark information, and representation landmark information, and the representation landmark information may be estimated in step S230 using Equation 11. Therefore, the identity landmarks can be calculated as follows:

number

[0042] FIG. 17 is a diagram illustrating a method for calculating a transformation matrix according to an embodiment of the present invention. Referring to FIG. 17, an artificial neural network receives an input image of a face of an arbitrary person. The artificial neural network may be a part of a known artificial neural network, but in one embodiment, the artificial neural network may be a ResNet. ResNet is a type of CNN (Convolutional Neural Network), and the present invention is not limited to a specific type of artificial neural network. MLP (Multi-Layer Perceptron) is a type of artificial neural network that stacks multiple layers of Perceptrons to overcome the limitations of single-layer Perceptrons. Referring to Figure 17, the MLP receives the output of the artificial neural network and landmark information corresponding to the face image as input. The MLP also outputs a transformation matrix. In FIG. 17, the artificial neural network and the MLP may be understood as collectively constituting one trained artificial neural network. Once the transformation matrix is ​​estimated through the trained artificial neural network, expression landmark information and identity landmark information can be calculated as described with reference to Figure 16. The landmark separation method according to the present invention can also be applied when there are only a very small number of face images or only one frame of face images.

[0043] The trained artificial neural network is trained to estimate low-dimensional eigenvectors and transformation coefficients from a large number of face images and their corresponding landmark information, and the trained artificial neural network is capable of estimating the eigenvectors and transformation coefficients even when only one frame of face image is given. Separating expression and identity landmarks for any person in this way can improve the quality of facial image processing techniques such as facial landmark-based face reconstruction, face classification, and face morphing. Face replay technology is a technique that synthesizes facial images and photographs that, given a target face and a driver face, mimic the movements of the driver's face but have the identity of the target face. Face morphing technology is a technique for synthesizing a facial image or photo of a third person with the characteristics of Person 1 and Person 2, given facial images or photos of Person 1 and Person 2. Traditional morphing algorithms find face key points and then divide the face into non-overlapping triangles or rectangles based on these key points. The photos of Person 1 and Person 2 are then aligned to synthesize a photo of the third person. However, because the positions of the key points of Person 1 and Person 2 are different, pixel-wise alignment of the photos of Person 1 and Person 2 to generate a photo of the third person can be quite unnatural. Known face morphing technologies do not distinguish between the subject's appearance and emotional characteristics, such as facial expressions, and therefore can produce low-quality morphed results. The landmark separation method according to the present invention can separate expression landmark information and identity landmark information from a single landmark information, thereby contributing to improving the results of face image processing techniques that utilize face landmarks. In particular, the landmark separation method according to the present invention is very useful because it can separate landmarks even when only a very small amount of face image data is provided.

[0044] 18 is a diagram illustrating a schematic configuration of a landmark separating apparatus according to an embodiment of the present invention. Referring to FIG. 18, a landmark separating apparatus 5000 according to an embodiment of the present invention includes a receiving unit 5100, a transformation matrix estimating unit 5200, and a computing unit 5300. The receiving unit 5100 receives a face image of a first person and landmark information corresponding to the face image. Here, the landmarks may be understood as a concept including main elements of a face as the face landmarks, such as the eyes, eyebrows, nose, mouth, and jaw line. The landmark information may also include information about the position, size, or shape of the major elements of the face, and may also include information about the color or texture of the major elements of the face. The first person refers to any person, and the receiving unit 5100 receives a face image of the person and landmark information corresponding to the face image. The landmark information is obtained using known technology, and any known method may be used. The present invention is not limited by the method of acquiring the landmarks. The transformation matrix estimation unit 5200 estimates a transformation matrix corresponding to the landmark information. The transformation matrix, together with a predetermined unit vector, can form the landmark information. For example, the first landmark information may be calculated by multiplying the unit vector by the first transformation matrix. Also, the second landmark information may be calculated by multiplying the unit vector by the second transformation matrix. The transformation matrix converts high-dimensional landmark information into low-dimensional data and may be used in Principal Component Analysis (PCA). PCA is a dimension reduction method that converts variables in a high-dimensional space into variables in a low-dimensional space by searching for new mutually orthogonal axes while maximally preserving the variance of the data. PCA first finds a hyperplane that is closest to the data, and then projects the data onto the low-dimensional hyperplane to reduce the dimension of the data. In PCA, the unit vector defining the i-th axis is taken as the i-th principal component (PC), and high-dimensional data can be converted to low-dimensional data by linearly combining these axes.

[0045] As mentioned above, the unit vectors, i.e., the principal components, may be determined in advance. Therefore, when new landmark information is received, a corresponding transformation matrix may be determined. In this case, multiple transformation matrices may exist for one landmark information. On the other hand, the transformation matrix estimation unit 5200 may use a learning model trained to estimate the above transformation matrix. The learning model may be understood as a model trained to estimate a PCA transformation matrix from an arbitrary face image and landmark information corresponding to the arbitrary face image. The learning model may be trained to estimate the transformation matrix from facial images of different people and landmark information corresponding to each facial image. There may be multiple transformation matrices corresponding to one piece of high-dimensional landmark information, but the learning model may be trained to output only one of the multiple transformation matrices. The landmark information used as input to the learning model may be obtained using a known method of extracting landmarks from a face image and visualizing them. Therefore, the transformation matrix estimation unit 5200 receives the face image of the first person and landmark information corresponding to the face image as input, and estimates and outputs one transformation matrix therefrom. On the other hand, the learning model may be trained to classify landmark information into multiple semantic groups corresponding to the right eye, left eye, nose, and mouth, and to output PCA transform coefficients corresponding to each of the multiple semantic groups. In this case, the semantic groups are not necessarily classified to correspond to the right eye, left eye, nose, and mouth, but may be classified to correspond to eyebrows, eyes, nose, mouth, and jaw line, or may be classified to correspond to eyebrows, right eye, left eye, nose, mouth, jaw line, ears, etc. The transformation matrix estimation unit 5200 may classify the landmark information into semantic groups of subdivided units according to the learning model, and estimate PCA transformation coefficients corresponding to the classified semantic groups.

[0046] The calculation unit 5300 calculates representation landmarks of the first person using the transformation matrix, and calculates identity landmarks of the first person using the representation landmarks. The landmark information may be separated into a plurality of sub-landmark information, for example, average landmark information, identity landmark information, and representation landmark information. In other words, the landmark information for a specific person in a specific frame may be represented as the sum of the average landmark information for all people's faces, the identity landmark information for the specific person only, and the facial expression and movement information for the specific person in the specific frame. The average landmark information can be defined by the following formula, and may be calculated based on many images that can be collected in advance. The learning model may be trained to estimate the PCA coefficient α(c, t) using a photo x(c, t) of a person c whose landmark information is to be separated and landmark information l(c, t) as input, as shown in Equation 8. Through such training, the learning model may estimate a PCA coefficient from an image of a specific person and its corresponding landmark information, or may estimate the low-dimensional eigenvector. When applying a trained neural network, the photo x(c', t) of the person c' for which landmark separation is to be performed and the landmark information l(c', t) are input to the neural network, and the PCA transformation matrix is ​​estimated. exp are the predicted (estimated) PCA coefficients and b exp The expression landmarks may be estimated as shown in Equation 11 using the above. On the other hand, as described with reference to Equation 8, the landmark information may be defined as the sum of the average landmark information, the identity landmark information, and the representation landmark information, and the representation landmark information may be estimated using Equation 11 in step S230. Therefore, the identity landmarks can be calculated as shown in Equation 12. Given a face image of any person, landmark information can be obtained from it, and expression landmark information and identity landmark information can be calculated from the face image and landmark information.

[0047] 19 is a diagram illustrating an example of a method for recreating a face using the present invention. Referring to FIG. 19, a target image 4100 and a driver image 4200 are shown, and the target image 4100 may replay an image corresponding to the driver image 4200. It can be seen that the re-enacted image 4300 has the characteristics of the target image 4100, but its facial expression corresponds to that of the driver image 4200. That is, the re-enacted image 4300 has the identity landmarks of the target image 4100, but the expression landmarks have characteristics corresponding to those of the driver image 4200. Therefore, it is important to properly separate identity landmarks and expression landmarks from a single landmark for natural facial reproduction.

[0048] 20 is a diagram schematically illustrating an environment in which an image transformation device and an image transformation method according to the present invention operate. Referring to FIG. 20, the environment in which a first terminal 6000 and a second terminal 7000 operate may include a server 10000, and the first terminal 6000 and the second terminal 7000 connected to the server 10000. For ease of explanation, only two terminals, i.e., the first terminal 6000 and the second terminal 7000, are shown in FIG. 20, but two or more terminals may be included. The explanations regarding the first terminal 6000 and the second terminal 7000 may apply to additional terminals, unless otherwise noted. The server 10000 may be connected to a communication network. The server 10000 may be connected to other external devices via the communication network. The server 10000 may transmit data to the other devices connected to the server 10000, or may receive data from the other devices. The communication network connected to the server 10000 may include a wired communication network, a wireless communication network, or a combined communication network. The communication network may include a mobile communication network such as 3G, LTE, or LTE-A. The communication network may include a wired or wireless communication network such as Wi-Fi, UMTS / GPRS, or Ethernet. The communication network may include a local area network such as Magnetic Secure Transmission (MST), Radio Frequency Identification (RFID), Near Field Communication (NFC), ZigBee, Z-Wave, Bluetooth, Bluetooth Low Energy (BLE), or Infrared communication (IR). The communication network may include a local area network (LAN), a metropolitan area network (MAN), or a wide area network (WAN).

[0049] The server 10000 may receive data from at least one of the first terminal 6000 and the second terminal 7000. The server 10000 may perform a calculation using the data received from at least one of the first terminal 6000 and the second terminal 7000. The server 10000 may transmit the result of the calculation to at least one of the first terminal 6000 and the second terminal 7000. The server 10000 may receive an intermediation request from at least one of the first terminal 6000 and the second terminal 7000. The server 10000 may select a terminal to which the intermediation request is to be transmitted. For example, the server 10000 may select the first terminal 6000 and the second terminal 7000. The server 10000 may mediate a communication connection between the selected first terminal 6000 and second terminal 7000. For example, the server 10000 may mediate a video call connection or a text transmission / reception connection between the first terminal 6000 and the second terminal 7000. The server 10000 may transmit connection information related to the first terminal 6000 to the second terminal 7000, or may transmit connection information related to the second terminal 7000 to the first terminal 6000. The connection information regarding the first terminal 6000 may include, for example, the IP address and port number of the first terminal 6000. Upon receiving the connection information regarding the second terminal 7000, the first terminal 6000 may attempt to connect to the second terminal 7000 using the received connection information.

[0050] A video call session between the first terminal 6000 and the second terminal 7000 may be established by a successful attempt to connect the first terminal 6000 to the second terminal 7000 or a successful attempt to connect the second terminal 7000 to the first terminal 6000. Through the video call session, the first terminal 6000 may transmit images and sounds to the second terminal 7000. The first terminal 6000 may encode the images and sounds into digital signals and transmit the encoded results to the second terminal 7000. Furthermore, through the video call session, the first terminal 6000 may receive images and sounds from the second terminal 7000. The first terminal 6000 may receive images and sounds encoded into digital signals and decode the received images and sounds. Through the video call session, the second terminal 7000 may transmit images and sounds to the first terminal 6000. Also, through the video call session, the second terminal 7000 may receive images and sounds from the first terminal 6000. In this way, the user of the first terminal 6000 and the user of the second terminal 7000 can make a video call with each other. The first terminal 6000 and the second terminal 7000 may be, for example, a desktop computer, a laptop computer, a smartphone, a smart tablet, a smart watch, a mobile terminal, a digital camera, a wearable device, or a portable electronic device. The first terminal 6000 and the second terminal 7000 may execute a program or an application. The first terminal 6000 and the second terminal 7000 may be the same type of device or different types of devices.

[0051] FIG. 21 is a flowchart illustrating an image deformation method according to an embodiment of the present invention. Referring to FIG. 21, an image deformation method according to an embodiment of the present invention includes a step of acquiring user facial landmark information (S2100), a step of generating a user feature map (S2200), a step of generating a target feature map (S2300), a step of generating a mixed feature map (S2400), and a step of generating a reenacted image (S2500). In step S2100, landmark information is acquired from a facial image of a user. The landmarks refer to facial features that characterize the user's face, and may include, for example, the user's eyes, eyebrows, nose, mouth, ears, or jawline. The landmark information may also include information about the position, size, or shape of major elements of the user's face. Furthermore, the landmark information may also include information about the color or texture of major elements of the user's face. The user may refer to any user using a terminal on which the image transformation method of the present invention is executed. In step S2100, a facial image of the user is received, and landmark information corresponding to the facial image is acquired. The landmark information is acquired using known techniques, and any known method may be used. The method of acquiring the landmark information does not limit the present invention. In step S2200, a transformation matrix corresponding to the landmark information may be estimated. The transformation matrix may constitute the landmark information together with a predetermined unit vector. For example, the first landmark information may be calculated by multiplying the unit vector by the first transformation matrix. Also, the second landmark information may be calculated by multiplying the unit vector by the second transformation matrix.

[0052] The transformation matrix converts high-dimensional landmark information into low-dimensional data and may be used in Principal Component Analysis (PCA). PCA is a dimension reduction method that converts variables in a high-dimensional space into variables in a low-dimensional space by searching for new mutually orthogonal axes while maximally preserving the variance of the data. PCA first finds a hyperplane that is closest to the data, and then projects the data onto the low-dimensional hyperplane to reduce the dimension of the data. In PCA, the unit vector defining the i-th axis is taken as the i-th principal component (PC), and high-dimensional data can be converted to low-dimensional data by linearly combining these axes.

number

[0053] The learning model may be trained to estimate the transformation matrix from facial images of different people and landmark information corresponding to each facial image. There may be multiple transformation matrices corresponding to one piece of high-dimensional landmark information, but the learning model may be trained to output only one of the multiple transformation matrices. The landmark information used as input to the learning model may be obtained using a known method of extracting landmarks from a face image and visualizing them. Therefore, in step S2100, the face image of the user and landmark information corresponding to the face image are received as input, and a transformation matrix is ​​estimated and output therefrom. On the other hand, the learning model may be trained to classify landmark information into multiple semantic groups corresponding to the right eye, left eye, nose, and mouth, and to output PCA transform coefficients corresponding to each of the multiple semantic groups. In this case, the semantic groups are not necessarily classified to correspond to the right eye, left eye, nose, and mouth, but may be classified to correspond to eyebrows, eyes, nose, mouth, jaw line, or may be classified to correspond to eyebrows, right eye, left eye, nose, mouth, jaw line, ears, etc. In step S2100, the landmark information may be classified into semantic groups in subdivided units according to the learning model, and PCA transformation coefficients corresponding to the classified semantic groups may be estimated.

[0054] Meanwhile, the user's expression landmarks are calculated using the transformation matrix. Landmark information can be decomposed into a plurality of sub-landmark information, but in the present invention, the landmark information is represented as follows:

number

number

number

[0055] The learning model may be trained to estimate the PCA coefficient α(c, t) using a photo x(c, t) of a person c whose landmark information is to be separated and landmark information l(c, t) as input, as shown in Equation 14. Through such training, the learning model may estimate a PCA coefficient from an image of a specific person and its corresponding landmark information, or may estimate the low-dimensional eigenvector. When applying a trained neural network, the photo x(c', t) of the person c' for which landmark separation is to be performed and the landmark information l(c', t) are input to the neural network, and the PCA transformation matrix is ​​estimated. exp are the predicted (estimated) PCA coefficients and b exp , we may use it to estimate representation landmarks as follows:

number

number

[0056] The user feature map generated in step S2200 includes information representing the characteristics of the facial expressions and facial movements of the user. The artificial neural network used in step S2200 may be a convolutional neural network (CNN), but various types of artificial neural networks may also be used. In step S2300, a target face image is received, and a target feature map and a pose-normalized target feature map are generated from style information and pose information corresponding to the target face image. The target refers to a person who is transformed by the present invention, and the user and the target may be different people, but are not necessarily limited to this. A reenacted image generated as a result of implementing the present invention may be transformed from a facial image of the target and displayed as the target imitating or copying the movements and expressions of the user. The target feature map includes information representing the facial expressions expressed by the target and the characteristics of the facial movements of the target. The pose-normalized target feature map may correspond to the output of an artificial neural network for the input style information, or the pose-normalized target feature map may include information corresponding to distinctive facial features of the target, excluding pose information of the target.

[0057] The artificial neural network used in step S2300 may be a CNN, similar to the artificial neural network used in step S2200, but the structure of the artificial neural network used in step S2200 may be different from the structure of the artificial neural network used in step S2300. The style information means information indicating a person's unique features in their face, and may include, for example, innate features appearing on the target's face, the size, shape, and position of landmarks, etc. Alternatively, the style information may include at least one of texture information, color information, and shape information corresponding to the target's facial image.

[0058] The target feature map may be understood as including data corresponding to expression landmark information obtained from facial images of the target, and the pose-normalized target feature map may be understood as including data corresponding to identity landmark information obtained from facial images of the target. Alternatively, the style information may include at least one of texture information, color information, and shape information corresponding to the facial image of the target. The mixed feature map may be generated such that the target landmarks have pose information corresponding to the user's landmarks. The artificial neural network used in step S2400 may be a CNN, similar to the artificial neural networks used in steps S2200 and S2300, and the structure of the artificial neural network used in step S2400 may be different from the structure of the artificial neural network used in the previous steps.

[0059] In step S2500, the mixed feature map and the pose-normalized target feature map are used to generate a reconstructed image for the target face image. As described above, the pose-normalized target feature map includes data corresponding to identity landmark information obtained from the facial image of the target, but the identity landmark information means information corresponding to the unique features of a person, regardless of expression information corresponding to the person's movement information or facial expression information. If the target's movements can be obtained to naturally follow the user's movements through the mixed feature map generated in step S2400, the unique characteristics of the target can be reflected in step S2500, achieving an effect similar to that of an actual target moving and expressing its own facial expressions.

[0060] 22 is a diagram illustrating an example of the results of performing an image transformation method according to an embodiment of the present invention, showing a target image, a user image, and a reenacted image, where the reenacted image has the facial movements and expressions of the user while preserving the facial features of the target. Comparing the target image and the replayed image in Figure 22, we can see that the two images show the same person, with only differences in facial expression. The eyes, nose, mouth, and hairstyle of the target image are the same as those of the replayed image, respectively. Meanwhile, the facial expression of the person in the reproduced image is substantially similar to the facial expression of the user. For example, if the user has their mouth open in the image of the user, the reproduced image will have an image of the target with their mouth open. Also, if the user has their head turned right or left in the image of the user, the reproduced image will have an image of the target turning their head right or left. When a real-time changing image of a user is received and a replayed image is generated based on the received image, the replayed image can change the target image in response to the real-time changing movements and facial expressions of the user.

[0061] 23 is a diagram illustrating the configuration of an image transformation apparatus according to an embodiment of the present invention. Referring to FIG. 23, an image transformation apparatus 8000 according to an embodiment of the present invention includes a landmark acquisition unit 8100, a first encoder 8200, a second encoder 8300, a blender 8400, and a decoder 8500. The landmark acquisition unit 8100 receives facial images of a user and a target, and acquires landmark information from each facial image. The landmarks refer to facial features that characterize the user's face, and may include, for example, the user's eyes, eyebrows, nose, mouth, ears, or jawline. The landmark information may also include information about the position, size, or shape of the user's major facial features. Furthermore, the landmark information may also include information about the color or texture of the user's major facial features. The user may refer to any user who uses a terminal on which the image transformation method according to the present invention is executed. The landmark acquisition unit 8100 receives a facial image of the user and acquires landmark information corresponding to the facial image. The landmark information is acquired using known technology, and any known method may be used. The present invention is not limited by the method of acquiring the landmark information.

[0062] The landmark acquisition unit 8100 may estimate a transformation matrix corresponding to the landmark information. The transformation matrix may constitute the landmark information together with a predetermined unit vector. For example, the first landmark information may be calculated by multiplying the unit vector by the first transformation matrix. Furthermore, the second landmark information may be calculated by multiplying the unit vector by the second transformation matrix. The transformation matrix converts high-dimensional landmark information into low-dimensional data and may be used in Principal Component Analysis (PCA). PCA is a dimension reduction method that converts variables in a high-dimensional space into variables in a low-dimensional space by searching for new mutually orthogonal axes while maximally preserving the variance of the data. PCA first finds a hyperplane that is closest to the data, and then projects the data onto the low-dimensional hyperplane to reduce the dimension of the data. In PCA, the unit vector defining the i-th axis is taken as the i-th principal component (PC), and high-dimensional data can be converted to low-dimensional data by linearly combining these axes.

[0063] On the other hand, the landmark acquisition unit 8100 may use a learning model trained to estimate the above transformation matrix. The learning model may be understood as a model trained to estimate a PCA transformation matrix from an arbitrary face image and landmark information corresponding to the arbitrary face image. The learning model may be trained to estimate the transformation matrix from facial images of different people and landmark information corresponding to each facial image. There may be multiple transformation matrices corresponding to one piece of high-dimensional landmark information, but the learning model may be trained to output only one of the multiple transformation matrices. The landmark information used as input to the learning model may be obtained using a known method of extracting landmarks from a face image and visualizing them. Therefore, the landmark acquisition unit 8100 receives the face image of the user and landmark information corresponding to the face image as input, and estimates and outputs one transformation matrix therefrom. On the other hand, the learning model may be trained to classify landmark information into multiple semantic groups corresponding to the right eye, left eye, nose, and mouth, and to output PCA transform coefficients corresponding to each of the multiple semantic groups. In this case, the semantic groups are not necessarily classified to correspond to the right eye, left eye, nose, and mouth, but may be classified to correspond to the eyebrows, eyes, nose, mouth, and jaw line, or may be classified to correspond to the eyebrows, right eye, left eye, nose, mouth, jaw line, ears, etc. The landmark acquisition unit 8100 may classify the landmark information into semantic groups in subdivided units according to the learning model, and estimate PCA transformation coefficients corresponding to the classified semantic groups.

[0064] Meanwhile, the transformation matrix may be used to calculate the user's expression landmarks. The landmark information may be separated into multiple sub-landmark information, but in the present invention, the landmark information is defined as the sum of the average human face landmark information, the person's individual specific face landmark information, and the person's expression face landmark information. In other words, the landmark information for a specific person in a specific frame may be represented as the sum of the average landmark information for all people's faces, the identity landmark information for the specific person only, and the facial expression and movement information for the specific person in the specific frame. Meanwhile, the expression landmarks correspond to pose information of the user's facial image, and the identity landmarks correspond to style information of the target's facial image. In summary, the landmark acquisition unit 8100 may receive the facial image of the user and the facial image of the target, and generate multiple pieces of landmark information therefrom, including respective expression landmark information and identity landmark information. The first encoder 8200 generates a user feature map from pose information of the user's facial image. The pose information corresponds to the expression landmark information and may include movement information and facial expression information of the facial image. The first encoder 8200 may also input the pose information corresponding to the user's facial image into an artificial neural network to generate the user feature map.

[0065] The user feature map generated by the first encoder 8200 includes information representing the characteristics of the facial expressions and facial movements of the user. The artificial neural network used in the first encoder 8200 may be a convolutional neural network (CNN), but various types of artificial neural networks may also be used. The second encoder 8300 generates a target feature map and a pose-normalized target feature map from the style information and pose information of the target face image. The target refers to a person who is transformed by the present invention, and the user and the target may be different people, but are not necessarily limited to this. A reenacted image generated as a result of implementing the present invention may be transformed from a facial image of the target and displayed as the target imitating or copying the movements and expressions of the user. The target feature map generated by the second encoder 8300 may be understood as data corresponding to the user feature map generated by the first encoder 8200, and includes information representing the facial expressions expressed by the target and the characteristics of the target's facial movements. The pose-normalized target feature map may correspond to the output of an artificial neural network for the input style information, or the pose-normalized target feature map may include information corresponding to distinctive facial features of the target, excluding pose information of the target.

[0066] The artificial neural network used in the second encoder 8300 may be a CNN, similar to the artificial neural network used in the first encoder 8200, but the structure of the artificial neural network used in the first encoder 8200 and the structure of the artificial neural network used in the second encoder 8300 may be different from each other. The style information means information indicating a person's unique features in their face, and may include, for example, innate features appearing on the target's face, the size, shape, and position of landmarks, etc. Alternatively, the style information may include at least one of texture information, color information, and shape information corresponding to the target's facial image. The target feature map may be understood as including data corresponding to expression landmark information obtained from facial images of the target, and the pose-normalized target feature map may be understood as including data corresponding to identity landmark information obtained from facial images of the target. Blender 8400 may generate a mixed feature map using the user feature map and the target feature map, and input pose information of the user's facial image and style information of the target's facial image into an artificial neural network to generate the mixed feature map.

[0067] The mixed feature map may be generated such that the target's landmarks have pose information corresponding to the user's landmarks. The artificial neural network used in Blender 8400 may be a CNN, similar to the artificial neural networks used in first encoder 8200 and second encoder 8300, and the structure of the artificial neural network used in Blender 8400 may be different from the structure of the artificial neural network used in first encoder 8200 or second encoder 8300. The user feature map and the target feature map input to the blender 8400 contain the user's facial landmark information and the target's facial landmark information, respectively, and generate a target face corresponding to the user's facial movements and expressions, and may perform an operation to match the user's facial landmarks with the target's facial landmarks so as to maintain the unique features of the target face. For example, in order to control the facial movement of the target in accordance with the facial movement of the user, landmarks such as the user's eyes, eyebrows, nose, mouth, jaw line, etc. may be understood as being linked to landmarks such as the eyes, eyebrows, nose, mouth, jaw line, etc. of the target, respectively.

[0068] Alternatively, the user's landmarks such as the eyes, eyebrows, nose, mouth, jaw line, etc. may be linked to the target's landmarks such as the eyes, eyebrows, nose, mouth, jaw line, etc., in order to control the target's facial expressions in line with the user's facial expressions. The decoder 8500 uses the mixed feature map and the pose-normalized target feature map to generate a reconstructed image for the target face image. As described above, the pose-normalized target feature map includes data corresponding to identity landmark information obtained from the facial image of the target, but the identity landmark information means information corresponding to the unique features of a person, regardless of expression information corresponding to the person's movement information or facial expression information. If the target's movements can be obtained to naturally follow the user's movements through the mixed feature map generated by Blender 8400, the unique characteristics of the target can be reflected in the decoder 8500, resulting in an effect similar to that of a real target moving and expressing its own facial expressions.

[0069] FIG. 24 is a diagram illustrating a schematic configuration of a landmark acquisition unit according to an embodiment of the present invention. Referring to FIG. 24, the landmark acquisition unit according to an embodiment of the present invention may include an artificial neural network, which receives a person's face image as an input. The artificial neural network may be a part of a known artificial neural network, but in one embodiment, the artificial neural network may be a ResNet. ResNet is a type of CNN (Convolutional Neural Network), and the present invention is not limited to a specific type of artificial neural network. An MLP (Multi-Layer Perceptron) is a type of artificial neural network that stacks multiple layers of Perceptrons to overcome the limitations of a single-layer Perceptron. Referring to FIG. 24, the MLP receives the output of the artificial neural network and landmark information corresponding to the face image as input. The MLP also outputs a transformation matrix. The artificial neural network and the MLP may be considered to constitute a single trained artificial neural network as a whole. Once the transformation matrix is ​​estimated through the trained artificial neural network, expression landmark information and identity landmark information can be calculated as described with reference to Figure 23. The image transformation device according to the present invention can also be applied to cases where there are only a very small number of facial images or only one frame of facial images. The trained artificial neural network is trained to estimate low-dimensional eigenvectors and transformation coefficients from a large number of face images and their corresponding landmark information, and the trained artificial neural network is capable of estimating the eigenvectors and transformation coefficients even when only one frame of face image is given. By separating expression landmarks and identity landmarks for any person in this way, the quality of facial image processing techniques such as facial landmark-based face reenactment, face classification, and face morphing can be improved.

[0070] FIG. 25 is a diagram schematically illustrating a configuration of a second encoder according to an embodiment of the present invention. 25, the second encoder 8300 according to an embodiment of the present invention may adopt a U-Net structure. U-Net refers to a U-shaped network, which basically performs a subdivision function and has a symmetrical structure. f y denotes the normalized flow map used to normalize the target feature map, and T denotes the warping function that performs the warping. j、 j=1....n y denotes the target feature map encoded in each convolutional layer. The second encoder 8300 receives the rendered target landmarks and target image as input and then encodes the target feature map and normalized flow map f y Also, the generated target feature map S jand the normalized flow map f y takes as input and performs a warping function to generate a warped target feature map. The warped target feature map here may be understood as being similar to the pose-normalized target feature map described above. Therefore, the warping function T may be understood as a function that generates data consisting of only the style information, i.e., identity landmark information, of the target, excluding the expression landmark information of the target. FIG. 26 is a diagram showing a schematic diagram of a blender structure according to one embodiment of the present invention. As described above, the Blender 8400 generates a mixed feature map from a user feature map and a target feature map, but the pose information of the user's facial image and the style information of the target's facial image may also be input to an artificial neural network to generate the mixed feature map.

[0071] 26 shows one user feature map and three target feature maps, but the number of target feature maps may be one, two, or more than three. Also, the small area inside each feature map shown in FIG. 25 represents information for an arbitrary landmark, and all of them show information for the same landmark. The user feature map and the target feature map input to the blender 8400 contain the user's facial landmark information and the target's facial landmark information, respectively, and generate a target face corresponding to the user's facial movements and expressions, and may perform an operation to match the user's facial landmarks with the target's facial landmarks so as to maintain the unique features of the target face. For example, in order to control the facial movement of the target in accordance with the facial movement of the user, landmarks such as the user's eyes, eyebrows, nose, mouth, jaw line, etc. may be understood as being linked to landmarks such as the eyes, eyebrows, nose, mouth, jaw line, etc. of the target, respectively. Alternatively, the user's landmarks such as the eyes, eyebrows, nose, mouth, jaw line, etc. may be linked to the target's landmarks such as the eyes, eyebrows, nose, mouth, jaw line, etc., in order to control the target's facial expressions in line with the user's facial expressions. Also, for example, the user feature map may be searched for eyes, and then the target feature map may be searched for eyes, and a mixed feature map may be generated such that the eyes in the target feature map follow the eye movements in the user feature map. Substantially the same operations may be performed by blender 8400 for other landmarks.

[0072] FIG. 27 is a diagram illustrating the structure of a decoder according to one embodiment of the present invention. Referring to FIG. 27, a decoder 8500 according to one embodiment of the present invention combines the pose-normalized target feature map generated by the second encoder 8300 and the mixed feature map z generated by the blender 8400. xy By using the input, the user's expression landmark information is applied to the target image. In FIG. 27, the data input to each block of the decoder 8500 is the pose-normalized target feature map generated by the second encoder 8300, and f u means a flow map that applies user expression landmark information to the pose-normalized target feature map. Additionally, the Warp-alignment block of the decoder 8500 performs a warping function using the output u of the previous block of the decoder 8500 and the pose-normalized target feature map as input. The warping function performed by the decoder 8500 is different from the warping function performed by the second encoder 8300, as it is intended to generate a reenacted image that mimics the user's movement and pose while preserving the unique features of the target. On the other hand, moving images can be generated based on the embodiments described above with reference to Figures 1 to 27. For example, as described above with reference to Figures 5a to 6b, it is possible to generate moving images by converting input still images. Alternatively, an input still image may be converted into a moving image based on an image transformation template. The image transformation template may include multiple frames, and each frame may be a still image. For example, multiple intermediate images (i.e., multiple still images) may be generated by applying each of the multiple frames to the input still image. Furthermore, the generated intermediate images may be combined to generate a moving image.

[0073] Alternatively, the input moving image may be converted to generate the moving image, in which case each of the first still images (frames) included in the input moving image is converted into a second still image, and the second still images are combined to generate the moving image. The embodiments described above with reference to FIGS. 1 to 27 may be realized by the content described below. For example, at least a portion of the content described below may be applied to at least one of the embodiments described above with reference to FIGS. 1 to 27. Furthermore, if the meaning of a term described below is identical or similar to the meaning of a term described with reference to FIGS. 1 to 27, they may be understood as terms referring to the same component. Furthermore, if the content described below is identical or similar to the content described above with reference to FIGS. 1 to 27, they may be understood as the same content. Furthermore, the content described below may be included in the content of the paper "MarioNETte: Few-Shot Face Reenactment Preserving Identity of Unseen Targets." When mismatches occur between the target identity and the driver identity, the quality of the results significantly degrades during face reconstruction, especially when capturing multiple images. Identity preservation issues, i.e., the model losing target details leading to output defects, are the most common failure modes. This issue has several potential causes, including leaking the driver's identity due to identity mismatches and unseen large pose processing.

[0074] To overcome these issues, we propose three components: an image attention block, a target feature alignment block, and a landmark transformation block. By processing and warping relevant features, the proposed architecture, called MarioNETte, achieves high-quality reconstruction of unseen identities in multi-imaging settings. Furthermore, the landmark transformation block decomposes landmarks to separate representational geometry, significantly mitigating the identity preservation problem. Comprehensive experiments are conducted to verify that the proposed framework surpasses all other criteria and generates highly realistic faces, even when the facial features between the target and driver are significantly inconsistent. Given a target face and a driver face, face replay aims to synthesize a face animated by the driver's movements while preserving the target's identity. Previously, generative adversarial networks (GANs) have been widely used, which have achieved great success in image generation. Xu et al. (2017; Wu et al. (2018) utilized CycleGAN (Zhu et al., 2017) to achieve high-fidelity face reconstruction results. However, CycleGAN-based approaches require at least several minutes of training data for each target and can only reconstruct predefined identities. This makes them less attractive in the real world, where reconstructing unseen targets is unavoidable. Therefore, face reconstruction approaches using multiple images attempt to reconstruct unseen targets using adaptive instance normalization (AdaIN) (Zakharov et al., 2019) or warping modules (Wiles, Koepke, and Zisserman 2018; Siarohin et al., 2019). However, current state-of-the-art methods suffer from a so-called identity preservation problem, where reconstruction is flawed by the inability to preserve the identity of the target. This problem is exacerbated when the driver's identity and the target's identity are different.

[0075] Examples of faulty and successful facial reconstructions generated by previous approaches and the proposed model are shown in Figures 28A-28C, respectively. In most cases, the failure of previous approaches can be divided into three modes: 1. Without considering identity mismatch, the driver's identity inhibits face synthesis, and the generated face resembles the driver (Figure 28A). 2. If the compressed vector representation (e.g., AdaIN layer) is insufficient to preserve the information of the target identity, the generated face may lack detailed features (Figure 28B). 3. The warping process introduces defects when handling large poses (Figure 28C). We propose a framework called MarioNETte, which aims to recreate the face of an unseen target across multiple imaging methods while preserving its identity without fine-tuning. We use an image attention block and target feature alignment, which allows MarioNETte to directly inject features from the target when generating images. We also propose a novel landmark transformation module, which reconciles identity inconsistencies in an unsupervised manner, further mitigating the identity preservation problem. This is described in detail below. We propose a multi-image face reconstruction framework called MarioNETte, which preserves target identity even when the driver's facial features differ significantly from those of the target. The proposed method uses image interest blocks, which allow the model to process the relevant locations of the target feature map, combined with target feature alignment, which involves multiple feature-level warping operations, to improve the quality of face reconstruction under different identities. We present a new landmark transformation method that accommodates different facial features of different people. The proposed method adapts the driver's landmarks to the target landmarks in an unsupervised manner, mitigating the identity preservation problem without additional label data. We proceed by comparing state-of-the-art methods using the VoxCeleb1 (Nagrani, Chung, and Zisserman 2017) and CelebV (Wu et al. 2018) datasets when the target identity and driver identity are consistent and distinct. These experiments, including user studies, demonstrate that the proposed method outperforms state-of-the-art methods. MarioNETte structure

[0076] Figure 29 shows the overall structure of the proposed model. The conditional generator G is a set of the driver x and the target image A reconstructed face is generated based on JPEG2025023956000039.jpg934, and a classifier D predicts whether the image is real or not. The generator consists of the following components: The preprocessor P extracts facial keypoints using a 3D landmark detector (Bulat and Tzimiropoulos 2017) and renders landmark images to correspond to the driver and target inputs, respectively. JPEG2025023956000040.jpg626 and JPEG2025023956000041.jpg827 is computed. The proposed landmark transformer is included in the preprocessor. It utilizes 3D landmarks instead of 2D landmarks, as it normalizes the size, translation, and rotation of landmarks before using them in the landmark transformer. Driver Encoder JPEG2025023956000042.jpg818 extracts pose and expression information from the driver input and creates a feature map of the driver z x Generate. Target Encoder JPEG2025023956000043.jpg723 employs a U-Net structure to extract style information from the target input and generate a warped target feature map. JPEG2025023956000044.jpg85 along with the target feature map z y Generate. Brenda JPEG2025023956000045.jpg943 is the driver feature map z x and target feature map JPEG2025023956000046.jpg844 is received and mixed feature map z xyThe proposed image attention block is the basic building block of Blender. ·decoder JPEG2025023956000047.jpg845 is the warped target feature map JPEG2025023956000048.jpg75 and mixed feature map z xy The decoder utilizes the proposed target feature alignment to synthesize the replayed image. Image Attention Block To transfer target style information to the driver, previous studies have encoded target information into vectors and then blended them with driver features via concatenation or AdaIN layers (Liu et al., 2019; Zakharov et al., 2019). However, encoding targets as spatially agnostic vectors results in the loss of target spatial information. Furthermore, these methods lack unique designs for multiple target images, so they use summary statistics (e.g., mean or maximum) to handle multiple targets, potentially losing target details.

[0077] To solve the aforementioned problems, we propose an image attention block (Figure 30). The proposed attention block is inspired by the encoder-decoder attention in transforms (VaSwani et al. 2017), where the driver feature map plays the role of attention query and the target feature map plays the role of attention memory. The proposed attention block uses multiple target feature maps (i.e., Z y ), the appropriate location of each feature (red box in Figure 30) is processed. Driver Feature Map JPEG2025023956000049.jpg638 and target feature map Considering JPEG2025023956000050.jpg782, the above attention is calculated as follows:

number

number

[0078] Target Feature Alignment Fine details of target identity can be preserved through warping of low-level features (Siarohin et al. 2019). Unlike previous approaches that calculate the difference between target and driver keypoints and estimate a warping flow map or affine transformation matrix (Balakrishnan et al. 2018; Siarohin et al. 2018; Siarohin et al. 2019), our proposed target feature alignment warps the target feature map in two stages: (1) target pose normalization, which generates a pose-normalized target feature map, and (2) driver pose adaptation, which aligns the fully qualified target feature map to the driver's pose (Figure 31). Through this two-stage process, the model can better handle the structural differences between different identities. Details are as follows: 1. Target Pose Normalization Target Encoder E y Feature maps encoded in JPEG2025023956000056.jpg829 is the estimated normalized flow map f y and warping function JPEG2025023956000057.jpg876 (1 in Figure 31) The next warp alignment block in the decoder is target pose agnostic. Process JPEG2025023956000059.jpg96. 2. Driver Pose Adaptation The warp alignment block in the decoder JPEG2025023956000060.jpg1135 and the output u of the previous block of the decoder. In some imaging settings, other target images (e.g., We average the resolution-compatible feature maps (JPEG2025023956000061.jpg1148). To apply the pose-normalized feature maps to the driver pose, we use a 1x1 convolution with u as input and driver f u Generate an estimated flow map of Alignment is performed using JPEG2025023956000062.jpg1130 (2 in Figure 31). After that, the above result is concatenated with u and input to the next remaining upsampling block.

[0079] Landmark conversion section Large structural differences between two facial landmarks significantly degrade the quality of the replay. Common approaches to these problems are to learn a transformation for every identity (Wu et al., 2018) or to prepare paired landmark data with similar representations (Zhang et al., 2019). However, such methods are unnatural in a multi-image setting where unseen identities are handled, and there are difficulties in collecting labeled data. To overcome these difficulties, we propose a novel landmark transformation module that transfers a driver's facial expression to an arbitrary target identity. The landmark transformation module is trained in an unsupervised manner, leveraging multiple videos of unlabeled human faces. Landmark Separation When viewing a video of a different identity, let x(c, t) be the t-th frame of the c-th video, and let l(c, t) be the 3D facial landmarks. First, we normalize the size, translation, and rotation of all landmarks, and then we define the normalized landmarks as Convert to JPEG2025023956000063.jpg1019. Inspired by 3D morphable face models (Blanz and Vetter 1999), we consider that normalized landmarks can be separated as follows:

number

number

[0080] Landmark Decomposition To separate identity and representational geometry in a multi-image setting, we introduce a neural network that regresses linear-based coefficients. Previously, such approaches have been widely used to model complex facial geometry (Blanz and Better 1999). We extract representational bases from training data by separating representational landmarks into semantic groups of faces (e.g., mouth, nose, eyes) and performing PCA on each group.

number

number

number

[0081] Experimental setup Dataset We trained the model and baseline using VoxCeleb1 (Nagrani, Chung, and Zisserman 2017), which contains 256x256 videos of 1,251 distinct identities. We used test splits of VoxCeleb1 and CelebV (Wu et al. 2018) to evaluate self-repeat and replay under other identities, respectively. We generated a test set by sampling a set of 2,083 images from 100 randomly selected videos in the VoxCeleb1 test split, and a set of 2,000 images uniformly sampled from all identities in CelebV. The CelebV data contains videos of five celebrities with various characteristics and is used to evaluate the model's performance in replaying targets that do not appear similar to real-world scenarios. Details of the loss function and training method can be found in Supplementary Materials A3 and A4. standard The MarioNETte transformations with and without the landmark transformation (MarioNETte+LT and MarioNETte) are compared with the state-of-the-art model for face reconstruction several times. The details of each criterion are as follows: x2 Faces (Wiles, Koepke, and Zisserman 2018) x2 Faces uses direct image distortion. We use a pre-trained model provided by the inventors, trained on VoxCeleb1. Monkey-Net (Siarohin et al. 2019) Monkey-Net employs feature-level warping. The implementation provided by the inventors is used. Due to the structure of the method, Monkey-Net can only receive one original image. NeuralHead (Zakharov et al. 2019) NeuralHead utilizes AdaIN layers. Since there is no reference implementation, we have made an honest attempt to replicate the results. This implementation is a feedforward version of the model (NeuralHead-FF), which omits the meta-learning and fine-tuning stages. This is because we use a single model to handle multiple identities.

[0082] index To evaluate the quality of the generated images, we compare models based on the following metrics: structural similarity (SSIM) (Wang et al. 2004) and peak signal-to-noise ratio (PSNR), which assess the low-level similarity between the generated and real images. We also report masked SSIM (M-SSIM) and masked PSNR (M-PSNR), where the measurements are restricted to the face region. In the absence of actual images of the target face driven by different identities, the following metrics are more relevant: We evaluate the identity preservation quality using the cosine similarity (CSIM) of the embedding vectors generated by a pre-trained face recognition model (Deng et al. 2019). To examine the model's ability to properly reproduce the pose and driver's expression, we calculate the root mean square error (PRMSE) of the head pose angle and the ratio of similar facial action unit values ​​between the generated image and the driving image. We use OpenFace (Baltrusaitis et al. 2018) to calculate the pose angle and action unit values. Experimental results We compared models under self-replay and self-repeat with different identities, including user studies. Ablation experiments were also performed. All experiments were performed in both single-shot and multiple-shot settings, with one target image used for single-shot and eight target images used for multiple-shot settings. self-reenactment Figure 34 shows the evaluation results of the models under the self-replay setting for VoxCeleb1. MarioNETte outperforms the other models in all metrics in the multiple capture setting, and in the single capture setting, it outperforms the other models in all metrics except PSNR. However, MarioNETte performs best in M-PSNR, which means it performs better in the face region compared to the baseline. The low CSIM obtained with NeuralHead-FF is indirect evidence of the capacity shortfall of AdaIN-based methods.

[0083] Other identity reenactments Figure 35 shows the evaluation results for replaying other identities on CelebV, and Figure 33 shows images generated from the proposed method and benchmark. MarioNETte and MarioNETte+LT properly preserve target identity and outperform other CSIM models. The proposed method alleviates the identity preservation problem regardless of whether the driver is the same identity or not. NeuralHead-FF performs slightly better than MarioNETte in terms of PRMSE and AUCON, but NeuralHead-FF's low CSIM means it fails to preserve target identity. The landmark transformer significantly improves identity preservation, albeit with a slight decrease in PRMSE and AUCON. This decrease may be due to the PCA criterion for representation decomposition not being sufficiently diverse to encompass the entire representation space. Furthermore, decomposition of identity and representation itself is a significant issue, especially in single-shot imaging settings. User research Two types of user studies are conducted to evaluate the performance of the proposed model. Comparative Analysis: Considering three example images of the target and a driver image, two images generated by different models were displayed, and human evaluators were asked to select the higher-quality image. Users were asked to evaluate the quality of the images in terms of (1) identity preservation, (2) reproduction of the driver's pose and facial expression, and (3) photorealism. We report the win rate of the baseline model compared to the proposed model. User-reported scores are believed to better reflect the quality of other models than other indirect measurements. Realism Analysis: Similar to Zakharov et al.'s (2019) user study protocol, three photos of the same person were presented to human raters. Two of the three photos were from a video recording, and the remaining were generated by the model. Users were instructed to select the image that differed from the other two in terms of identity within a time limit of 3 seconds. We report the percentage of each model's identity preservation and photorealism. In both studies, 150 examples were sampled from CelebV and uniformly distributed among 100 different raters.

[0084] Figure 36 shows that this model is preferred over the conventional method, with a realism score that is significantly greater than the conventional method. Consequently, this demonstrates MarioNETte's ability to generate realistic reenactments while preserving target identity in terms of human perception. There is a slight preference for MarioNETte over MarioNETte+LT. This is because, as shown in Figure 35, MarioNETte+LT has a higher identity preservation ability despite a slight reduction in representational transfer. Because MarioNETte+LT's identity preservation ability surpasses all other models in realism score, i.e., nearly twice as high as MarioNETte's score in the multiple-image setting, the slight reduction in representational transfer is not a significant issue. Ablation experiment Ablation tests were performed to investigate the effectiveness of the proposed components. We compare the following configurations that replicate other identities while keeping everything else the same: (1) MarioNETte is the proposed method in which both image attention and target feature alignment are applied. (2) AdaIN corresponds to a model similar to MarioNETte, where the image attention block is replaced by the AdaIN remaining block and target feature alignment is omitted. (3) +Attention is MarioNETte with only the image attention block applied. (4) +Alignment is MarioNETte in which only target feature alignment is used. Figure 37 shows the results of the ablation test. Due to identity preservation (e.g., CSIM), AdaIN has difficulty combining style features that rely only on AdaIN remaining blocks. +Attention handles the appropriate coordinates, significantly mitigating the problem in both single-shot and multi-shot settings. +Alignment exhibits higher CSIM compared to +Attention, but struggles to generate plausible images for unseen poses and expressions, resulting in worse PRMSE and AUCON. MarioNETte leverages both attention and target feature alignment and outperforms +Alignment on all metrics under consideration.

[0085] +Alignment, which relies entirely on target feature alignment for replay, easily fails due to large pose differences between the target and the driver. MarioNETte overcomes this. When a single driver image is presented along with three target images (Figure 38A), +Alignment reveals a defect in the forehead (shown by the arrow in Figure 38B). This is due to (1) warping low-level features with a large pose input and (2) combining features from multiple targets with various poses. MarioNETte, on the other hand, handles the situation appropriately by processing not only the appropriate spatial coordinates within the target image but also the appropriate image among several target images. The attention map, which highlights the area the image attention block focuses on, is shown in white in Figure 38A. MarioNETte processes the forehead and appropriate target images (targets 2 and 3 in Figure 38A) that have a similar pose to the driver. Related technologies Classic approaches to face replay include explicit 3D modeling of human faces (Blanz and Vetter 1999), where the parameters of the driver and target 3D models are typically calculated from single images and then blended together (Thies et al. 2015; Thies et al. 2016). Image warping is another popular approach, which rectifies the target image using estimated flow obtained from a 3D model (Cao et al. 2013) or rare landmarks (Averbuch-Elor et al. 2017). Face replay research has embraced recent successes using neural networks that explore various image-to-image translation structures (Isola et al. 2017), such as the work of Xu et al. (2017) and Wu et al. (2018) combined with cycle consistency loss (Zhu et al. 2017). Hybridization of the two approaches has also been explored. Kim et al. (2018) trained an image translation network that maps replayed renderings of a 3D face model to a realistic output.

[0086] Recently, structures capable of fusing target style information with driver spatial information have been proposed. AdaIN (Huang and Belongie 2017; Huang et al. 2018; Liu et al. 2019) layers, attention mechanisms (Zhu et al. 2019; Lathuili`ere et al. 2019; Park and Lee 2019), deformation tasks (Siarohin et al. 2018; Dong et al. 2018), and GAN-based methods (Bao et al. 2018) have all been widely adopted. Similar ideas have been applied several times to the image-based face replay setting, including the use of AdaIN layers combined with image-level (Wiles, Koepke, and Zisserman 2018) and feature-level (Siarohin et al. 2019) warping and meta-learning (Zakharov et al. 2019). The problem of identity mismatch has been studied using methods such as CycleGAN-based landmark transformation (Wu et al. 2018) and landmark swapper (Zhang et al. 2019). Although effective, these methods require separate models for each person or datasets containing image pairs, which are difficult to obtain. conclusion Here, we propose a framework for face replay. The proposed image attention block and target feature alignment, along with the landmark transformation unit, can handle identity mismatches that occur using other people's landmarks. The proposed method does not require an additional fine-tuning step for identity adaptation, significantly increasing the usefulness of the model in real-world deployments. Experiments, including human evaluation, suggest the superiority of the proposed method. Future research directions include improving the landmark transformation part and better handling landmark decomposition to make the replay more convincing. Supplementary Materials Detailed information on MarioNETte structure

[0087] structural design Driver image x and K target image Given JPEG2025023956000086.jpg1033, the proposed face reconstruction framework, called MarioNETte, first generates a 2D landmark image (i.e., JPEG2025023956000087.jpg610 and Generates JPEG2025023956000088.jpg832. 3D landmark detector Using JPEG2025023956000089.jpg864 (Bulat and Tzimiropoulos 2017), JPEG2025023956000090.jpg936 and The facial key points are extracted from the image, which contains information about the pose and expression shown in JPEG2025023956000091.jpg835. After that, the 3D landmarks are rasterized into an image using Rasterizer R. I get JPEG2025023956000092.jpg967. A simple rasterizer is used to orthogonally project 3D landmark points (e.g., (x, y, z)) onto a 2D xY plane (e.g., (x, y)), and the projected landmarks are grouped into eight categories: left eye, right eye, contour, nose, left eyebrow, right eyebrow, inner mouth, and outer mouth. Lines are drawn between the points in a predefined order using predefined colors for each group (e.g., red, red, green, blue, yellow, yellow, cyan, and cyan, respectively). This results in the rasterized image shown in Figure 39. MarioNETte is a conditional image generator JPEG2025023956000093.jpg987 and projection classifier JPEG2025023956000094.jpg828. The classifier D is JPEG2025023956000095.jpg77 is the rasterized landmark Determine whether the image is a real image of the distribution of data taking into account the conditional input of JPEG2025023956000096.jpg87 and identity c.

[0088] generator JPEG2025023956000097.jpg987 is further broken down into four components: the target encoder, the driver encoder, the blender, and the decoder. JPEG2025023956000098.jpg928 is the target feature map warped from the target image. Target feature map z encoded along with JPEG2025023956000099.jpg95 y Generate the driver encoder JPEG2025023956000100.jpg821 receives the driver image and creates the driver feature map z x Generate Blender JPEG2025023956000101.jpg952 combines the encoded feature maps and generates the mixed feature map z xy Generates a decoder. JPEG2025023956000102.jpg1055 generates the reconstructed image. y are concatenated per channel and fed to the target encoder. Target Encoder JPEG2025023956000103.jpg928 employs a U-Net (Ronneberger, Fischer, and Brox 2015) style structure, which includes five downsampling blocks and four upsampling blocks using skip connections. The five feature maps generated by the downsampling blocks are The most downsampled feature map, s5, in JPEG2025023956000104.jpg931 is the encoded target feature map z y Used as the rest JPEG2025023956000105.jpg930 is converted to a normalized feature map. Normalized flow map JPEG2025023956000106.jpg959 has the following warping features: Using JPEG2025023956000107.jpg88, each feature map is normalized as a feature map. Convert to JPEG2025023956000108.jpg1145.

number

[0089] We employ a bilinear sampler-based warping function, which is widely used with neural networks due to its differentiation potential (Jaderberg et al. 2015; Balakrishnan et al. 2018; Siarohin et al. 2019). Each s j Since the width and height are different, f y The size of S j To match the size of y applies to. Driver Encoder JPEG2025023956000110.jpg921 is composed of the four remaining downsampling blocks and contains the image of the driver's landmarks. x Taking the driver's feature map z x Generate. Brenda JPEG2025023956000111.jpg853 is z x Location information and target style feature map z y and the mixed feature map z xyを Generate. Stack attention blocks on three images to create a blender. decoder JPEG2025023956000112.jpg957 is composed of four warp alignment blocks and the remaining upsampling blocks, the last of which is followed by an additional convolutional layer and a hyperbolic tangent activation function. Classifier JPEG2025023956000113.jpg829 consists of five remaining downsampled blocks without a self-attention layer. We employ a projection classifier with a slight modification that removes the global summation layer from the original structure. By removing the global summation layer, the classifier generates scores for multiple patches similar to the PatchGAN classifier (Isola et al. 2017). The network is constructed using the remaining upsampling and downsampling blocks proposed by Brock, Donahue, and Simonyan (2019). All batch normalization layers are replaced with instance normalization, except for the target encoder and classifier, which do not have normalization layers. ReLU is used as the activation function. The number of channels through which the output is downsampled (or upsampled) is doubled (or halved). The minimum number of channels is set to 64, and the maximum number of channels is set to 512 for all layers. The input image used as input for the target encoder, driver encoder, and classifier is first projected through a convolutional layer to match the channel size of 64.

[0090] Position Encoding We use a slightly modified sinusoidal position encoding introduced by Vaswani et al. (2017). First, we divide the number of channels for position encoding in half. Then, we use half of these to encode horizontal coordinates and the remaining half to encode vertical coordinates. To encode relative positions, we normalize the absolute coordinates by the width and height of the feature map. Thus, Given the feature map JPEG2025023956000114.jpg844, the corresponding position encoding JPEG2025023956000115.jpg847 is calculated as follows:

number

[0091] loss function The model was adversarially trained using a projection classifier D (Miyato and Koyama 2018). The classifier aims to distinguish between real images of identity c and synthetic images of c generated by G. Since paired target and driver images of different identities cannot be obtained without explicit annotation, the model was trained using target and driver images extracted from the same video sequence. Thus, x and y i The identity of is always the same for all target and driver image pairs during training (e.g., c). That is, JPEG2025023956000117.jpg838). We use the hinge GAN loss (Lim and Ye 2017) to optimize the discriminator D as follows:

number

number

number

[0092] Learning details To stabilize the adversarial learning, we apply spectral normalization (Miyato et al. 2018) to all layers of the classifier and generator. We also use the convex hull of facial landmarks as a face region mask, and calculate the perceptual loss by weighting the mask position by three times. We train the model using the Adam Optimizer, where we use a 2 × 10 -4 A learning rate of 5×10 is used for the classifier. -5 is used for the generator and style encoder. Unlike the setting in Brock, Donahue, and Simonyan (2019), we update the discriminator only once per generator update. During training,、 λ P 10, λ PF 0.01, λ FM Set the number of target images,K,to 10 and 4, respectively. Detailed information on landmark conversion Landmark Separation Formally, the landmark separation is calculated as follows:

number

[0093] Landmark Decomposition Using the representation geometry obtained from the VoxCeleb1 training data, we evaluate the representation criterion b exp To calculate, the landmarks are divided into other groups (e.g., left eye, right eye, eyebrow, mouth, etc.) and PCA is performed for each group. PCA dimensions of 8, 8, 8, 16, and 8 are used for each group, resulting in a total of 48 expression criteria n exp get. The landmark decomposition part is trained separately on the VoxCeleb1 training set. Before training the landmark decomposition part, the parameters of each expression are normalized and the standard normal distribution is used to facilitate regression training. We followed the JPEG2025023956000135.jpg825. We used ResNet50 pre-trained on Imagenet (He et al. 2016) and extracted features from the first layer to the last layer just before the global average pooling layer. The extracted image features are then used as average landmarks. JPEG2025023956000136.jpg88 subtracted normalized landmarks The network is concatenated with JPEG2025023956000137.jpg85 and fed into a two-layer MLP, after which ReLU activation is performed. The whole network has a learning rate of 3 × 10 -4 The Adam optimizer is used to optimize the predicted and target representation parameters by minimizing the MSE loss between them. During training, gradient clipping was used, with the maximum gradient normalized to 1. The representation strength parameter λ exp is set to 1.5.

[0094] Additional ablation experiments Quantitative results 34 and 35, MarioNETte outperforms NeuralHead-FF in PRMSE and AUCON in the VoxCeleb1 self-replica setting, but this is reversed when replicating other identities in CelebV. We explain this phenomenon through ablation studies. Figure 40 shows the evaluation results of the ablation model under the self-replay setting for VoxCeleb1. Unlike the evaluation results for other identity replays on CelebV (Figure 37 in this paper), +Alignment and MarioNETte outperform AdaIN in terms of PRMSE and AUCON. This phenomenon may be due to the characteristics of the training dataset and other inductive biases of other models. VoxCeleb1 consists of short video clips (typically 5–10 seconds long) that exhibit similar poses and expressions between the driver and the target. Unlike AdaIN-based models, which do not recognize spatial information, the proposed image attention block and target feature alignment encode spatial information in the target image. This is likely why the proposed model can overfit to similar identity pairs with similar pose and expression settings.

[0095] Qualitative results Figures 43 and 44 show the results of the ablation model recreating other identities in CelebV under single and multiple imaging settings, respectively. While AdaIN is unable to generate images similar to the target identity, +Attention successfully preserves the key characteristics of the target. The Target Feature Alignment module adds details to the generated images. However, while MarioNETte produces more natural images in several imaging settings, +Alignment does not easily handle images of multiple targets with various poses and expressions. Inference time In this section, we report the inference time of our model. We measured the latency of the provided method while generating 256 × 256 images with different numbers of target images K ∈ {1, 8}. Each configuration was run 300 times and the average speed is reported. We used an Nvidia Titan Xp and Pytorch 1.0.1.post2. As mentioned in this paper, we extracted 3D facial landmarks using the open-source implementation of Bulat and Tzimiropoulos (2017). Figure 41 shows the inference time analysis of the model. The total inference time of the proposed models, MarioNETte+LT and MarioNETte, can be derived as shown in Figure 3. The z used to calculate the target encoding while generating the replay video y and JPEG2025023956000138.jpg98 is generated only once at the beginning. Therefore, it is inferred by dividing it into the target encoding part and the driver generated part. Because we perform batch inference on multiple target images, the inference time of the proposed components (e.g., target encoder and target landmark transformer) scales nonlinearly with the number of target images K. On the other hand, the open-source 3D landmark detector processes images sequentially, resulting in a linear scaling of processing time.

[0096] Additional examples of generated images We provide additional qualitative results for the baseline methods and proposed model on the VoxCeleb1 and CelebV datasets. Except for Monkey-Net, which is designed to use only a single image, we report qualitative results for the single-shot and multi-shot (8 target image) settings. In the multi-shot case, only one target image is displayed due to limited space. Figures 45 and 46 compare different methods for self-replay of VoxCeleb1 in single-shot and multiple-shot settings, respectively. Examples of single-shot and multiple-shot replays where the identity of the driver and target do not match in VoxCeleb1 are shown in Figures 13 and 48. Figures 49, 50, and 51 show qualitative results for the CelebV dataset. Various self-replay settings for single and multiple captures are compared in Figures 15 and 50. The results of replaying different identities from CelebV depending on the multiple capture setting can be seen in Figure 51. Figure 52 shows examples of failures generated by MarioNETte+LTd while replaying a single image under different identity settings in VoxCelebl. The main cause of failure appears to be the large pose difference between the driver and target. The above-described embodiments may also be implemented in the form of a recording medium containing computer-executable instructions, such as a program module executed by a computer. The computer-readable medium may be any available medium that can be accessed by a computer, and may include both volatile and nonvolatile media, and removable and non-removable media. Computer-readable media may also include computer storage media, which may include all volatile and nonvolatile, removable and non-removable media implemented by any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data.

[0097] At least one of the components, parts, modules, or units (collectively referred to in this paragraph as "components") represented by blocks in the drawings, such as those in FIGS. 2, 9, 17, 18, 23-27, and 29-32, may be implemented as various hardware, software, and / or firmware structures that perform the respective functions described above, according to exemplary embodiments. For example, at least one of these components may use direct circuitry, such as a memory, a processor, a logic circuit, a look-up table, or the like, that may perform the respective functions through control by one or more microprocessors or other control devices. At least one of these components may also be tangibly implemented as a module, a program, or part of code including one or more executable instructions for performing a particular logic function, and may be executed by one or more microprocessors or other control devices. At least one of these components may also be implemented by or include a processor, such as a central processing unit (CPU), that executes the respective function, microprocessor, or the like. Two or more of these components may be combined into a single component that performs all operations or functions of the two or more components combined. Furthermore, at least a portion of the functions of at least one of these components may be performed by other components of these components. Although the above block diagrams do not show buses, communication between components may occur via a bus. Functional aspects of the above exemplary embodiments may be implemented by algorithms executed by one or more processors. Furthermore, components represented by blocks or processing steps may use any number of related technologies, such as electronic configuration, signal processing and / or control, data processing, and the like. Although the embodiments of the present invention have been described above with reference to the accompanying drawings, those skilled in the art will understand that the present invention may be embodied in other specific forms without changing the technical spirit or essential characteristics thereof. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and are not limiting.

Claims

1. receiving a static image from a user; Obtaining at least one image transformation template; and converting the still image into a moving image using the obtained image conversion template.

2. The image conversion method according to claim 1 , wherein the motion picture includes a plurality of frames.

3. The image conversion method of claim 2 , wherein the plurality of frames includes a frame corresponding to the still image.

4. 3. The image conversion method according to claim 2, wherein the moving image has a frame corresponding to the still image as a first frame and a frame corresponding to the image conversion template as a last frame.

5. The image transformation method according to claim 1 , wherein the image transformation template is selected by the user from among a plurality of image transformation templates stored in a user terminal of the user.

6. receiving at least one reference image from a user; 2. The image transformation method according to claim 1, wherein in the step of obtaining the image transformation template, the reference image is obtained as the image transformation template.

7. The image conversion method according to claim 6 , wherein the reference image is an image of the user or an image of a different person selected by the user.

8. The step of converting the still image into a moving image includes: Extracting texture information of the user's face contained in the still image; Extracting face landmark information of a person included in the image transformation template; The image conversion method according to claim 1 , further comprising the step of combining the texture information and the landmark information to generate the moving image.

9. A computer-readable recording medium having a program recorded thereon for executing the method according to any one of claims 1 to 8.

10. an image receiving unit for receiving an image from a user; A template acquisition unit that acquires at least one image transformation template; and an image converter that converts a still image into a moving image by utilizing the at least one image conversion template acquired, The image conversion unit converts a static image received by the image receiving unit into the moving image.

11. The image conversion device according to claim 10 , wherein the moving image includes a plurality of frames.

12. The image conversion device according to claim 11 , wherein the plurality of frames includes a frame corresponding to the still image.

13. 12. The image conversion device according to claim 11, wherein the moving image has a frame corresponding to the still image as a first frame and a frame corresponding to the image conversion template as a last frame.

14. Further comprising a template storage unit for storing a plurality of image transformation templates; The image conversion device according to claim 10 , wherein the image conversion section converts the still image into the moving image using an image conversion template selected by the user from a plurality of image conversion templates stored in the template storage section.

15. The image receiving unit receives at least one reference image from the user; The image transformation device according to claim 10 , wherein the template acquisition unit acquires the reference image as the image transformation template.

16. The image conversion device according to claim 15 , wherein the reference image is an image of the user or an image of a different person selected by the user.

17. The image conversion unit extracts texture information of a user's face included in the still image, Extracting face landmark information of a person included in the image transformation template; The image conversion device according to claim 10 , wherein the moving image is generated by combining the texture information and the landmark information.

Images