Image conversion device and method, and computer-readable recording medium
The image conversion method using an artificial neural network addresses the lack of emotional consideration in conventional facial analysis by converting still images into natural-moving images, enhancing user experience and landmark data separation accuracy.
Patent Information
- Application Number
- JP2020185991
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-02-25
- Filing Date
- 2020-11-06
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-11-06
AI Technical Summary
Conventional facial image analysis and utilization techniques fail to consider facial appearance features and emotional characteristics, leading to decreased performance in processing facial landmarks.
An image conversion method using an artificial neural network to convert still images into natural-looking moving images by extracting and combining texture and landmark information, allowing for accurate separation of landmark data and facial expressions.
The method provides a moving image that mimics a user's facial expressions without direct capture, enhancing user experience and improving the accuracy of landmark data separation.
Smart Images

Figure 0007579674000138 
Figure 0007579674000139 
Figure 0007579674000140
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to Patent Application No. 10-2019-0141723, filed on November 7, 2019, Patent Application No. 10-2019-0177946, filed on December 30, 2019, Patent Application No. 10-2019-0179927, filed on December 31, 2019, and Patent Application No. 10-2020-0022795, filed on February 25, 2020, all of which are incorporated herein by reference in their entireties. The present invention relates to an image conversion device and method, and a computer-readable recording medium. More specifically, the present invention relates to an image conversion device and method capable of converting a still image into a natural moving image, and a computer-readable recording medium. The present invention relates to a landmark data separating device and method, and a computer-readable recording medium. More specifically, the present invention relates to a landmark data separating device and method, and a computer-readable recording medium, which can more accurately separate landmark data from a face included in an image. The present invention relates to a landmark separation device and method, and a computer-readable recording medium. More specifically, the present invention relates to a landmark separation device and method capable of separating landmarks from one frame or a small number of frames, and a computer-readable recording medium. The present invention relates to an image transformation device and method, and a computer-readable recording medium. More specifically, the present invention relates to an image transformation device and method capable of generating images that naturally transform according to the characteristics of different images, and a computer-readable recording medium. [Background technology]
[0002] Most portable personal devices have a built-in camera and may capture static images and moving images such as videos. When a user of a portable personal device needs a moving image of a desired facial expression, the user must capture the moving image with the camera built into the portable personal device. If the moving image is not captured with the desired facial expression, the user must repeat the capture until a satisfactory result is obtained. Therefore, a method is needed that can replace the desired facial expression with a still image input by the user and convert it into a natural moving image. Techniques for analyzing and utilizing human facial images based on facial landmarks obtained by extracting facial key points are being actively researched. Facial landmarks include the results of extracting the origins of major elements of a face, such as the eyes, eyebrows, nose, mouth, and jaw line, or extracting contour lines by connecting these points. Facial landmarks are mainly used in techniques such as facial expression classification, pose analysis, and face synthesis and deformation. However, conventional face image analysis and utilization techniques based on face landmarks do not take into account facial appearance features and emotion-related characteristics when processing face landmarks, which leads to poor performance. Therefore, in order to improve the performance of face image analysis and utilization techniques, there is a need to develop a technique for separating face landmarks including emotion-related characteristics. Summary of the Invention [Problem to be solved by the invention]
[0003] An object of the present invention is to provide an image conversion device and method capable of converting still images into natural moving images, and a computer-readable recording medium. An object of the present invention is to provide a landmark data separation device and method, and a computer-readable recording medium, that can more accurately and precisely separate landmark data from a face included in an image. An object of the present invention is to provide a landmark separation device and method, and a computer-readable recording medium, that are capable of separating landmarks even when the amount of data is small. The present invention aims to provide an image transformation device and method, as well as a computer-readable recording medium, that, when given a target image to be transformed, can use an image of a user different from the target image to generate an image that matches the user's image but has the characteristics of the target image. [Means for solving the problem]
[0004] An image conversion method according to one embodiment of the present invention is an image conversion method using an artificial neural network, and includes the steps of receiving a static image from a user, obtaining at least one image conversion template, and converting the still image into a moving image using the obtained image conversion template. Effect of the Invention
[0005] The present invention may provide an image conversion device and method, as well as a computer-readable recording medium, that provide moving images having the same effect as moving images captured by a user directly changing their facial expressions, without the need to directly capture the moving images. The present invention may provide an image conversion device and method, as well as a computer-readable recording medium, that provide a user with a moving image generated by converting a still image, thereby providing an enjoyable user experience. The present invention may provide a landmark data separation device and method, as well as a computer-readable recording medium, that can separate landmark data more accurately and precisely from a face included in an image. The present invention may provide a landmark data separation device and method, as well as a computer-readable recording medium, capable of separating landmark data that more accurately contains information about facial characteristics and facial expressions contained in an image. The present invention may provide a landmark separation device and method, and a computer-readable recording medium, capable of separating landmarks even in an object with a small amount of data. The present invention may provide an image transformation device and method, as well as a computer-readable recording medium, that, when given a target image to be transformed, can use an image of a user that is different from the target image to generate an image that matches the user's image but has the characteristics of the target image. [Brief description of the drawings]
[0006] [Figure 1] FIG. 1 is a diagram showing a schematic diagram of an environment in which the image conversion method according to the present invention is executed. [Diagram 2] FIG. 2 is a diagram showing an outline of the configuration of an image conversion device according to an embodiment of the present invention. [Diagram 3] FIG. 3 is a flow chart illustrating an image conversion method according to an embodiment of the present invention. [Figure 4] FIG. 4 is a diagram illustrating an example of an image conversion template according to an embodiment of the present invention. [Figure 5a] FIG. 5a is an exemplary diagram illustrating a process for generating a moving image according to one embodiment of the present invention. [Figure 5b] FIG. 5b is a diagram illustrating an example of a generated moving image according to an embodiment of the present invention. [Figure 6a] FIG. 6a is a diagram illustrating an example of a process for generating a moving image according to another embodiment of the present invention. [Figure 6b] FIG. 6b is a diagram illustrating an example of a generated moving image according to another embodiment of the present invention. [Figure 7] FIG. 7 is a diagram showing an outline of the configuration of an image conversion device according to an embodiment of the present invention. [Figure 8] FIG. 8 is a schematic diagram illustrating an environment in which the method for extracting landmark data from a face contained in an image according to the present invention is carried out. [Figure 9] FIG. 9 is a diagram illustrating a schematic configuration of a landmark data separation device according to an embodiment of the present invention. [Figure 10] FIG. 10 is a diagram for explaining a method for extracting face landmark data according to an embodiment of the present invention. [Figure 11] FIG. 11 is a flow chart illustrating a method for extracting various types of landmark data according to one embodiment of the present invention. [Figure 12] FIG. 12 is a diagram illustrating an example process of converting facial expressions contained in an image according to another embodiment of the present invention. [Figure 13] FIG. 13 is a comparison table illustrating the effect of converting facial expressions contained in an image using the landmark data separation method according to the present invention. [Figure 14] FIG. 14 is a diagram illustrating a schematic configuration of a landmark data separation device according to an embodiment of the present invention. [Figure 15] FIG. 15 is a diagram illustrating an environment in which the landmark separation device according to the present invention operates. [Figure 16] FIG. 16 is a flow chart that illustrates a landmark separation method according to one embodiment of the present invention. [Figure 17] FIG. 17 is a diagram illustrating a method for calculating a transformation matrix according to an embodiment of the present invention. [Figure 18] FIG. 18 is a diagram showing a schematic configuration of a landmark separating device according to one embodiment of the present invention. [Figure 19] FIG. 19 is an exemplary diagram showing a method for recreating a face using the present invention. [Figure 20] FIG. 20 is a diagram illustrating an environment in which the image transformation device and image transformation method according to the present invention operate. [Figure 21] FIG. 21 is a flow chart that illustrates an image deformation method according to an embodiment of the present invention. [Figure 22] FIG. 22 is a diagram illustrating an example result of performing an image transformation method according to an embodiment of the present invention. [Diagram 23] FIG. 23 is a diagram showing an outline of the configuration of an image transformation device according to an embodiment of the present invention. [Figure 24] FIG. 24 is a diagram illustrating a schematic configuration of a landmark acquisition unit according to an embodiment of the present invention. [Diagram 25] FIG. 25 is a diagram illustrating a schematic configuration of a second encoder according to an embodiment of the present invention. [Figure 26] FIG. 26 is a schematic diagram showing the structure of a blender according to one embodiment of the present invention. [Figure 27] FIG. 27 is a diagram illustrating the structure of a decoder according to one embodiment of the present invention. [Fig. 28a-c] Figures 28a-c show examples of identity preservation failure and the improvement produced by the proposed method: Figure 28a shows the interference of the driver shape, Figure 28b shows the loss of target identity details, and Figure 28c shows the warping failure for large poses. [Figure 29] FIG. 29 shows the overall structure of MarioNETte. [Diagram 30] FIG. 30 shows the structure of an image attention block. [Diagram 31] FIG. 31 shows the structure of the target feature alignment. [Diagram 32] FIG. 32 shows the structure of the landmark decomposition section. [Diagram 33] Figure 33 shows images generated by the proposed method, the baseline, and replaying different identities on CelebV under a single imaging setting. [Diagram 34] Figure 34 shows the evaluation results of VoxCeleb1 in a self-repeat setting. [Diagram 35] Figure 35 shows the evaluation results of recreating different identities on CelebV. [Diagram 36]Figure 36 shows the results of a user study in which users re-enacted different identities on CelebV. [Figure 37] FIG. 37 shows a comparison of ablation models for recapitulating different identities in CelebV. [Figure 38] Figure 38a shows the driver and target images superimposed with the attention map, and Figure 38b shows the failure case of +Alignment and the improved result produced by MarioNETte. [Figure 39] FIG. 39 shows an example of rasterized facial landmarks. [Diagram 40] Figure 40 shows a comparison of ablation models for the self-repeat setting on the VoxCeleb1 dataset. [Diagram 41] Figure 41 shows the inference speed of each component of the model. [Diagram 42] Figure 42 shows the inference speed of the overall model for generating a single image from K target images. [Diagram 43] FIG. 43 shows qualitative results of a single-shot ablation model under different identity settings in CelebV. [Diagram 44] Figure 44 shows the qualitative results of the ablation model, replicated several times under different identity settings in CelebV. [Diagram 45] FIG. 45 shows qualitative results of a single-imaging self-replay setup on VoxCeleb1. [Diagram 46] FIG. 46 shows qualitative results of the multiple image self-replay setup on VoxCeleb1. [Figure 47] FIG. 47 shows the qualitative results of single-shot replays under different identity settings on VoxCeleb1. [Figure 48] Figure 48 shows the qualitative results of re-imaging several times under different identity settings on VoxCeleb1. [Figure 49] FIG. 49 shows qualitative results of a single-imaging self-replay setup on CelebV. [Figure 50]Figure 50 shows qualitative results of a multiple imaging self-replay setup on CelebV. [Figure 51] Figure 51 shows the qualitative results of re-imaging several times under different identity settings in CelebV. [Figure 52] Figure 52 shows an example of a failure made by MarioNETte+LTd during a single replay under different identity settings in VoxCeleb1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0007] The advantages and features of the present invention, as well as the methods of achieving the same, will become apparent from the following detailed description of the embodiments in conjunction with the accompanying drawings. In this regard, the embodiments of the present invention may have various forms and are not limited to the description set forth herein. Rather, these embodiments will comprehensively understand the present disclosure and fully convey the scope of the present disclosure to those skilled in the art, and the present disclosure will be defined solely by the appended claims. The same reference numerals refer to the same elements throughout the specification. Although "first" or "second" is used to describe various components, these components are not limited to the above terms. The above terms may be used to distinguish one component from another component. Therefore, the first component described below may be the second component within the technical concept of the present invention. The terms used in the present specification are for describing the embodiments and are not intended to limit the present invention. In the present specification, the singular form includes the plural form unless otherwise specified in the text. The words "comprises" or "comprising" used in the specification do not exclude the presence or addition of one or more other components or steps in the components or steps mentioned. Unless otherwise defined, all terms used in this specification may be interpreted in a manner commonly understood by a person having ordinary skill in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless they are clearly and specifically defined.
[0008] Fig. 1 is a diagram showing an outline of an environment in which an image conversion method according to the present invention is executed. Referring to Fig. 1, the environment in which an image conversion method according to the present invention is executed may include a server 10 and a terminal 20 connected to the server 10. For convenience of explanation, only one terminal is shown in Fig. 1, but multiple terminals may be included. The explanation regarding the terminal 20 may be applied to a terminal that may be added, except for the explanation to be specifically mentioned. In an embodiment of the present invention, the server 10 may receive an image from the terminal 20, convert the received image into an arbitrary format, and then transmit the converted image to the terminal 20. Alternatively, the server 10 may function as a platform that provides a service that the terminal 20 may access and use. The terminal 20 may convert an image selected by a user of the terminal 20, and transmit the converted image to the server 10. The server 10 may be connected to a communication network. The server 10 may be connected to other external devices via the communication network. The server 10 may transmit data to the other devices connected to each other, or may receive data from the other devices. The communication network connected to the server 10 may include a wired communication network, a wireless communication network, or a combined communication network. The communication network may include a mobile communication network such as 3G, LTE, or LTE-A. The communication network may include a wired or wireless communication network such as Wi-Fi, UMTS / GPRS, or Ethernet. The communication network may include a local area network such as Magnetic Secure Transmission (MST), RFID (Radio Frequency Identification), NFC (Near Field Communication), Zigbee, Z-Wave, Bluetooth, Bluetooth Low Energy (BLE), or InfraRed communication (IR). The communication network may include a Local Area Network (LAN), a Metropolitan Area Network (MAN), or a Wide Area Network (WAN), etc. The server 10 may be connected to the terminal 20 via a communication network. When the server 10 is connected to the terminal 20, the server 10 may transmit and receive data to and from the terminal 20 via the communication network. The server 10 may execute any calculation using data received from the terminal 20. The server 10 may transmit the result of the calculation to the terminal 20. The terminal 20 may be a desktop computer, a laptop computer, a smart phone, a smart tablet, a smart watch, a mobile terminal, a digital camera, a wearable device, or a portable electronic device, etc. The terminal 20 may execute a program or an application.
[0009] FIG. 2 is a diagram showing an outline of the configuration of an image conversion device according to an embodiment of the present invention. 2, an image conversion device 100 according to an embodiment of the present invention includes an image receiving unit 110, a template acquiring unit 120, and an image conversion unit 130. The image conversion device 100 may be configured by the server 10 or the terminal 20 described with reference to FIG. 1. Therefore, each component included in the image conversion device 100 may also be configured by the server 10 or the terminal 20. The image receiving unit 110 receives an image from a user. The image may include the face of the user, and may be a still image or a static image. Meanwhile, the size of the face of the user included in the image may differ from image to image. For example, the size of the face included in image 1 may have a pixel size of 100×100, and the size of the face included in image 2 may have a pixel size of 200×200.
[0010] The image receiving unit 110 may extract only the face area from the image received from the user, and then provide this to the image converting unit 130 . The image receiving unit 110 may extract an area corresponding to the user's face from the image including the user's face, with a predetermined size. For example, if the predetermined size is 100×100 and the size of the area corresponding to the user's face included in the image is 200×200, the image receiving unit 110 may reduce the image with a size of 200×200 to 100×100 and then extract the image. Alternatively, a method of extracting an image with a size of 200×200 and then converting it to an image with a size of 100×100 may be used. The template acquisition unit 120 acquires at least one image transformation template. The image transformation template may be understood as a tool that can transform an image received by the image receiving unit 110 into a new image of a specific form. For example, when an image received by the image receiving unit 110 includes a neutral face of a user, a new image including a smiling face of the user can be generated by using a specific image transformation template. The image transformation template may be predetermined to any template or may be selected by a user. The image conversion unit 130 may receive a still image corresponding to the face region from the image receiving unit 110. Furthermore, the image conversion unit 130 may convert the still image into a moving image using an image conversion template acquired by the template acquisition unit 120.
[0011] FIG. 3 is a flow chart illustrating an image conversion method according to an embodiment of the present invention. Referring to FIG. 3, the image transformation method according to an embodiment of the present invention may include step S110 of receiving a still image, step S120 of obtaining an image transformation template, and step S130 of generating a moving image. The image conversion method according to the present invention is an image conversion method using an artificial neural network, and can acquire a still image in step S110. The still image may include a user's face, and may include one frame. In step S120, at least one image transformation template may be obtained from among the plurality of image transformation templates stored in the image transformation device 100. The image transformation template may be selected by the user from among the plurality of image transformation templates stored in the image transformation device 100. The image transformation template may be understood as a tool that can transform the image received in step S110 into a new image of a particular form. For example, when the image received in step S110 contains a neutral face of a user, using a particular image transformation template may generate a new image containing a smiling face of the user. In another embodiment, when the image received in step S110 contains a smiling face of the user, another specific image transformation template may be used to generate a new image containing an angry face of the user. In some embodiments, at least one reference image may be received from a user in step S120. For example, the reference image may be an image of the user or an image of another person selected by the user. If the user does not select one of a plurality of defined templates but selects a reference image, the reference image may be taken as the image transformation template. That is, the reference image may be understood to function similarly to the image transformation template. In step S130, the still image may be converted into a moving image using the acquired image conversion template. To convert the still image into a moving image, texture information may be extracted from the user's face included in the still image. The texture information may be color and visual texture information of the user's face.
[0012] In addition, to convert a still image into a moving image, landmark information may be extracted from an area corresponding to a person's face included in the image conversion template. The feature point information may be obtained from a specific shape, pattern, color, or combination thereof included in the person's face based on an image processing algorithm. In addition, the image processing algorithm may be, but is not limited to, any of SIFT (Scale Invariant Feature Transform), HOG (HiStogram of Oriented Gradient), Haar feature, Ferns, LBP (Local Binary Pattern), and MCT (Modified Census Transform). The video may be generated by combining the texture information and the landmark information. In some embodiments, the video may include a plurality of frames. The video may have a first frame corresponding to the still image and a last frame corresponding to the image transformation template. For example, the facial expression of the user included in the still image may be the same as the face included in the first frame of the video, and when the texture information and landmark information are combined, the facial expression of the user included in the still image may be transformed according to the landmark information, and the last frame of the video may include a frame corresponding to the transformed user's face. When generating a video using an artificial neural network, the video can gradually change from the facial expression of the user included in the still image to the facial expression of the user converted according to the landmark information, that is, at least one or more frames may be included between the first frame and the last frame of the video, and the facial expression included in each of the at least one or more frames may gradually change. In this way, by using an artificial neural network, it is possible to generate moving images that have the same effect as moving images captured by a user directly changing their facial expressions, without the need to directly capture the moving images.
[0013] FIG. 4 is a diagram illustrating an example of an image conversion template according to an embodiment of the present invention. A plurality of image conversion templates may be stored in the image conversion device 100. Each of the plurality of image conversion templates may include outline images corresponding to eyebrows, eyes, and a mouth. The plurality of image conversion templates may correspond to various facial expressions, such as a sad expression, a happy expression, a winking expression, a melancholic expression, a neutral expression, a surprised expression, and an angry expression, and each of the plurality of image conversion templates may include information on different facial expressions. The outline images corresponding to the various facial expressions are different from each other. Thus, each of the plurality of image conversion templates may include different outline images. Referring to FIG. 2, the image conversion unit 130 may extract landmark information from an outline image included in the image conversion template.
[0014] FIG. 5a is an exemplary diagram illustrating a process for generating a moving image according to one embodiment of the present invention. 4 and 5a, there are shown a still image 31, an image transformation template 32, and a moving image 33 generated using the still image 31 and the image transformation template 32. For example, the still image 31 may include a smiling face of a user. The image transformation template 32 may include an outline image corresponding to the eyebrows, eyes, and mouth of a winking and smiling face. On the other hand, although the video 33 shown in FIG. 5a is considered to include only one frame, the video 33 may be understood to indicate the last frame constituting the video generated by the image conversion unit 130 or step S130. The image conversion device 100 may extract texture information of an area corresponding to the user's face from the still image 31. The image conversion device 100 may also extract landmark information from the image conversion template 32. The image conversion device 100 may generate a moving image 33 by combining the texture information of the still image 31 and the landmark information of the image conversion template 32. The motion picture 33 is shown as a single image including the user's winking face, however the motion picture 33 includes multiple frames. A motion picture 33 including multiple frames will now be described with reference to Figure 5b. FIG. 5b is a diagram illustrating an example of a generated moving image according to an embodiment of the present invention. 5a and 5b, there may be at least one frame between the first frame 33_1 and the last frame 33_n of the video 33. For example, the still image 31 may correspond to the first frame 33_1 of the video 33. Also, the image including the winking face of the user may correspond to the last frame 33_n of the video 33. At least one frame between the first frame 33_1 and the last frame 33_n of the video 33 may each include an image of the user's face with the eyes gradually closing.
[0015] FIG. 6a is a diagram illustrating an example of a process for generating a moving image according to another embodiment of the present invention. 4 and 6a, a still image 41, a reference image 42, and a moving image 43 generated using the still image 41 and the reference image 42 are shown. For example, the still image 41 may include a smiling face of a user. The reference image 42 may include a face that is winking and smiling. The face included in the reference image 42 may be the face of a person different from the user. On the other hand, although the video 43 shown in FIG. 6a is considered to include only one frame, the video 43 may be understood to represent the last frame constituting the video generated by the image conversion unit 130 or step S130. The image conversion device 100 may extract texture information of an area corresponding to the user's face from the still image 41. The image conversion device 100 may also extract landmark information from the reference image 42. The image conversion device 100 may extract landmark information in areas corresponding to the eyebrows, eyes, and mouth of the face included in the reference image 42. The image conversion device 100 may generate a moving image 43 by combining the texture information of the still image 41 and the landmark information of the reference image 42. The motion picture 43 is shown as a single image including the user's smiling and winking face, however the motion picture 43 includes multiple frames. A motion picture 43 including multiple frames will now be described with reference to Figure 6b.
[0016] FIG. 6b is a diagram illustrating an example of a generated moving image according to another embodiment of the present invention. 6a and 6b, there may be at least one frame between the first frame 43_1 and the last frame 43_n of the video 43. For example, the still image 41 may correspond to the first frame 43_1 of the video 43. Also, an image including the user's smiling and winking face may correspond to the last frame 43_n of the video 43. At least one frame between the first frame 43_1 and the last frame 43_n of the video 43 may each include an image of the user's face with the eyes gradually closing and the mouth gradually opening.
[0017] FIG. 7 is a diagram showing an outline of the configuration of an image conversion device according to an embodiment of the present invention. 7, the image conversion device 200 may include a processor 210 and a memory 220. Those skilled in the art will appreciate that in addition to the components shown in FIG. 13, other typical components are also included. The image conversion device 200 may be similar to or identical to the image conversion device 100 shown in Fig. 2. The image receiving unit 110, the template acquiring unit 120, and the image conversion unit 130 included in the image conversion device 100 may be further included in a processor 210. The processor 210 controls all operations of the image conversion device 200 and may include at least one processor such as a CPU. The processor 210 may include at least one specialized processor corresponding to each function, or may be a single integrated processor. The memory 220 may store programs, data, or files related to the artificial neural network. The memory 220 may store instructions executable by the processor 210. The processor 210 may execute programs stored in the memory 220, read data and files stored in the memory 220, and store new data. The memory 220 may also store program instructions, data files, data structures, and the like, alone or in combination.
[0018] The processor 210 may obtain a still image from the input image. The still image may include the user's face and may include one frame. The processor 210 may read at least one image transformation template from among a plurality of image transformation templates stored in the memory 220. Alternatively, the processor 210 may read at least one reference image stored in the memory 220. For example, the at least one reference image may be input by a user. The reference image may be an image of the user or an image of another person selected by the user. If the user does not select one of a plurality of defined templates but selects a reference image, the reference image may be taken as the image transformation template. The processor 210 may convert the still image into a moving image using the acquired image conversion template. To convert the still image into a moving image, the processor 210 may extract texture information from the user's face included in the still image. The texture information may be color and visual texture information of the user's face. In addition, to convert a still image into a moving image, landmark information may be extracted from an area corresponding to a person's face included in the image conversion template. The feature point information may be obtained from a specific shape, pattern, color, or combination thereof included in the person's face based on an image processing algorithm. In addition, the image processing algorithm may be, but is not limited to, any of SIFT (Scale Invariant Feature Transform), HOG (Histogram of Oriented Gradient), Haar feature, Ferns, LBP (Local Binary Pattern), and MCT (Modified Census Transform).
[0019] The video may be generated by combining the texture information and the landmark information. The video may include a plurality of frames. The video may have a frame corresponding to the still image as a first frame and a frame corresponding to the image transformation template as a last frame. For example, the facial expression of the user included in the still image may be the same as the face included in the first frame included in the video. In addition, by combining the texture information and the landmark information, the facial expression of the user included in the still image may be transformed corresponding to the landmark information, and the last frame included in the video may include a frame corresponding to the transformed user's face. The video generated by the processor 210 may have a shape as shown in Figures 5b and 6b. The processor 210 may store the generated motion picture in memory 220 and output the motion picture so that it can be viewed by a user. As described with reference to Figures 1 to 7, when a user uploads a still image to the user's terminal 20, the image conversion device 200 may convert the still image into a moving image and provide it to the user. Even if the user does not directly capture a moving image, the user can be provided with a moving image having the same effect as a moving image captured while directly changing facial expressions. Moreover, the image conversion device 200 can provide a user with a pleasant user experience by providing the user with a moving image generated by converting the moving image into a still image.
[0020] FIG. 8 is a schematic diagram illustrating an environment in which the method for extracting landmark data from a face contained in an image according to the present invention is carried out. Referring to Fig. 8, an environment in which the method for extracting landmark data according to the present invention is executed may include a server 10-1 and a terminal 20-1 connected to the server 10-1. For convenience of explanation, only one terminal is shown in Fig. 8, but multiple terminals may be included. The explanation regarding the terminal 20-1 may be applied to terminals that may be added, except for explanations that should be specifically mentioned. In an embodiment of the present invention, server 10-1 may receive an image from terminal 20-1, extract landmark data from a face contained in the received image, calculate necessary data from the extracted landmark data, and then transmit the calculated data to terminal 20-1. Alternatively, the server 10-1 may function as a platform that provides a service that the terminal 20-1 may connect to and use. The terminal 20-1 may extract landmark data from a face included in an image, calculate necessary data from the extracted landmark data, and then transmit the calculated data to the server 10-1.
[0021] The server 10-1 may be connected to a communication network. The server 10-1 may be connected to other external devices via the communication network. The server 10-1 may transmit data to the other devices connected to each other, and may receive data from the other devices. The communication network connected to the server 10-1 may include a wired communication network, a wireless communication network, or a combined communication network. The communication network may include a mobile communication network such as 3G, LTE, or LTE-A. The communication network may include a wired or wireless communication network such as Wi-Fi, UMTS / GPRS, or Ethernet. The communication network may include a local area network such as Magnetic Secure Transmission (MST), RFID (Radio Frequency Identification), NFC (Near Field Communication), Zigbee, Z-Wave, Bluetooth, Bluetooth Low Energy (BLE), or InfraRed communication (IR). The communication network may include a Local Area Network (LAN), a Metropolitan Area Network (MAN), or a Wide Area Network (WAN), etc. The server 10-1 may be connected to the terminal 20-1 via a communication network. When the server 10-1 and the terminal 20-1 are connected to each other, the server 10-1 may transmit and receive data to and from the terminal 20-1 via the communication network. The server 10-1 may execute any calculation using data received from the terminal 20-1. The server 10-1 may transmit the result of the calculation to the terminal 20-1. The terminal 20-1 may be a desktop computer, a laptop computer, a smart phone, a smart tablet, a smart watch, a mobile terminal, a digital camera, a wearable device, or a portable electronic device, etc. The terminal 20-1 may execute a program or an application.
[0022] FIG. 9 is a diagram illustrating a schematic configuration of a landmark data separation device according to an embodiment of the present invention. 9, a landmark data separation device 100-1 according to an embodiment of the present invention may include an image receiving unit 110-1, a landmark data calculation unit 120-1, and a landmark data storage unit 130-1. The landmark data separation device 100-1 may be configured by the server 10-1 or the terminal 20-1 described with reference to Fig. 8. Therefore, each component included in the landmark data separation device 100-1 may also be configured by the server 10-1 or the terminal 20-1. The image receiving unit 110-1 may receive a plurality of images from a user. Each of the plurality of images may include only one person. That is, each of the plurality of images may include the face of one person, and the people included in the plurality of images may be different people. Image receiving section 110-1 may extract only the face area from each of the multiple images, and then provide the extracted face area to landmark data calculation section 120-1. The landmark data calculation unit 120-1 may calculate facial landmark data contained in each of the multiple images, average landmark data of all faces contained in the multiple images, characteristic landmark data of a specific face contained in a specific image among the multiple images, and facial expression landmark data of a specific face. In some embodiments, the landmark data may be the result of extraction of face key points. A method for extracting landmark data is described with reference to FIG.
[0023] FIG. 10 is a diagram for explaining a method for extracting face landmark data according to an embodiment of the present invention. The landmark data may be obtained by extracting the starting points of major features of the face, such as the eyes, eyebrows, nose, mouth, and jaw line, or by extracting contour lines drawn by connecting these points. The landmark data may be used in techniques such as facial expression classification, pose analysis, facial synthesis of different people, or facial deformation. 9 again, landmark data calculation unit 120-1 may calculate average landmark data of faces included in a plurality of images. The average landmark data may be a result of extracting an average human face shape. The landmark data calculation unit 120-1 may calculate landmark data from a specific image that includes a specific face among the multiple images. More specifically, the landmark data may be calculated from a specific face landmark data included in a specific frame among multiple frames included in the specific image. Furthermore, the landmark data calculation unit 120-1 may calculate characteristic landmark data of a specific face included in a specific image among the multiple images. The characteristic landmark data may be calculated based on face landmark data included in each of multiple frames included in the specific image. The data calculation unit 120-1 may also calculate average landmark data, landmark data for a specific frame, and characteristic landmark data to calculate facial expression landmark data for a specific frame in a specific image. For example, the facial expression landmark data may correspond to specific facial expressions and movement information of major elements such as the eyes, eyebrows, nose, mouth, and jaw line. The landmark data storage unit 130-1 may store data calculated by the landmark data calculation unit 120-1. For example, the landmark data storage unit 130-1 may store average landmark data, landmark data of a specific frame, characteristic landmark data, and facial expression landmark data, which are calculated by the landmark data calculation unit 120-1.
[0024] FIG. 11 is a flow chart illustrating a method for extracting various types of landmark data according to one embodiment of the present invention. 9 and 11, in step S1100, the landmark data separation device 100-1 may receive a plurality of images. Each of the plurality of images may include only one person. That is, each of the plurality of images may include the face of one person, and the people included in the plurality of images may be different people. In step S1200, the landmark data separation device 100-1 extracts the average landmark data I m The average landmark data I m can be expressed as follows:
number
[0025] For example, the landmark data for a particular frame I (c、t) may be main origin information of a specific face included in the t-th frame of the c-th image among the multiple images C. In other words, the specific image may be the c-th image, and the specific frame may be the t-th frame. In step S1400, the landmark data separation device 100-1 extracts specific facial characteristic landmark data I id(c) Characteristic landmark data I id(c) can be expressed as follows:
number
[0026] More specifically, the landmark data separation device 100-1 extracts specific facial expression landmark data I exp(c、t) The facial landmark data I exp(c、t) can be expressed as follows:
number
[0027] FIG. 12 is a diagram illustrating an example process of converting facial expressions contained in an image according to another embodiment of the present invention. 11 and 12, the server 10-1 or the terminal 20-1 receives the facial expression landmark data I separated from the landmark data separation device 100-1. exp(c、t) , Average landmark data I m , and characteristic landmark data I id(c) may be utilized to convert only the facial expression into the facial expression of the face contained in the first image 300 while maintaining the appearance of the face contained in the first image 300 . For example, the first image 300 is c x Among the frames in the th image, x The second image 400 may correspond to the cth frame among the multiple images. y Among the frames in the th image, y It may correspond to the cth frame. x The second image and c y The first and second images may be different images. The facial landmark data contained in the first image 300 may be separated as follows.
number
[0028] The facial landmark data contained in the second image 400 may be separated as follows.
number
number
[0029] FIG. 13 is a comparison table illustrating the effect of converting facial expressions contained in an image using the landmark data separation method according to the present invention. The MarioNETte model is a model that converts facial expressions contained in an image without using a landmark data separation method. When using the MarioNETte model, the naturalness of the converted image was measured to be 0.147. The MarioNETte+LT model is a model that converts facial expressions contained in an image using a landmark data separation method. When using the MarioNETte model, the naturalness of the converted image was measured to be 0.280. In other words, it was confirmed that the image converted using the MarioNETte+LT model is 1.9 times more natural than the image converted using the MarioNETte model.
[0030] FIG. 14 is a diagram illustrating a schematic configuration of a landmark data separation device according to an embodiment of the present invention. 14, the landmark data separation device 200-1 may include a processor 210-1 and a memory 220-1. A person having ordinary skill in the art related to this embodiment will understand that in addition to the components shown in FIG. 14, other general components are further included. The image conversion device 200-1 may be similar to or identical to the landmark data separation device 100-1 shown in Fig. 9. The image receiving unit 110-1 and the landmark data calculation unit 120-1 included in the landmark data separation device 100-1 may be included in the processor 210-1. The processor 210-1 controls the overall operation of the landmark data separation device 200-1 and may include at least one processor such as a CPU. The processor 210-1 may include at least one specialized processor corresponding to each function, or may be a single integrated processor. The memory 220-1 may store programs, data, or files that control the landmark data separation device 200-1. The memory 220-1 may store instructions executable by the processor 210-1. The processor 210-1 may execute programs stored in the memory 220-1, read data or files stored in the memory 220-1, or store new data. The memory 220-1 may also store program instructions, data files, data structures, and the like, either alone or in combination.
[0031] The processor 210-1 may receive a plurality of images, each of which may include only one person, i.e., each of the plurality of images may include the face of one person, and the people included in the plurality of images may be different people. The processor 210-1 may store the received images in the memory 220-1. The processor 210-1 calculates landmark data I of each face included in a plurality of images C. (c、t) The landmark data separation device 100-1 may calculate an average value of all the extracted landmark data. The calculated average value is referred to as average landmark data I m may also correspond to The processor 210-1 performs I / F on landmark data for a specific frame among a plurality of frames of a specific image including a specific face among the plurality of images. (c、t) may be calculated. Landmark data for a specific frame I (c、t) may be main origin information of a specific face included in the t-th frame of the c-th image among the multiple images C. In other words, the specific image may be the c-th image, and the specific frame may be the t-th frame. The processor 210-1 selects a specific facial characteristic landmark data I id(c)The c-th image may contain multiple frames that include various facial expressions of a particular face. Therefore, the characteristic landmark data I id(c) In order to calculate, the processor 210-1 uses specific facial expression landmark data I exp The average value of JPEG0007579674000020.jpg1541 may be set to 0. Therefore, characteristic landmark data I id(c) is a specific facial expression landmark data I exp The average value of The calculation may be performed without taking JPEG0007579674000021.jpg1541 into consideration. Feature Landmark Data I id(c) Calculate landmark data for each of the multiple frames included in the c-th image, and calculate the average landmark data of each of the multiple frames. Calculate JPEG0007579674000022.jpg1535 and calculate the average landmark data of the cth image Average landmark data from multiple images from JPEG0007579674000023.jpg1535 I m may be defined as the value minus
[0032] The processor 210-1 selects specific facial expression landmark data I exp(c、t) The facial landmark data I exp(c、t) may correspond to a specific facial expression included in the t-th frame and movement information of the eyes, eyebrows, nose, mouth, jaw line, etc. included in the specific face. More specifically, the facial landmark data I exp(c、t) is the landmark data for a specific frame. (c、t) Average landmark data from I m and characteristic landmark data I id(c) may be defined as the value minus The processor 210-1 receives the separated facial landmark data I exp(c、t) , Average landmark data Im , and characteristic landmark data I id(c) may be stored in memory 220-1. As will be described with reference to FIGS. 8 to 14, the landmark data separation devices 100-1 and 200-1 according to an embodiment of the present invention can separate more accurate and precise landmark data from a face included in an image. Furthermore, the landmark data separation devices 100-1 and 200-1 can separate landmark data that more accurately contains information related to the characteristics and facial expressions of the faces contained in the images. In addition, the server 10-1 or the terminal 20-1 including the landmark data separation device 100-1 or 200-1 receives the separated facial expression landmark data I exp(c、t) , Average landmark data I m , and characteristic landmark data I id(c) By utilizing this, it is possible to realize a technology that naturally converts facial expressions into facial expressions in a second image while maintaining the appearance of the face in a first image.
[0033] Fig. 15 is a diagram illustrating an environment in which the landmark separation device according to the present invention operates. Referring to Fig. 15, the environment in which the first terminal 2000 and the second terminal 3000 operate may include a server 1000, and the first terminal 2000 and the second terminal 3000 connected to the server 1000. For convenience of explanation, Fig. 15 shows only two terminals, i.e., the first terminal 2000 and the second terminal 3000, but two or more terminals may be included. For terminals that may be added, the explanation regarding the first terminal 2000 and the second terminal 3000 may be applied, except for the explanation to be specifically mentioned. The server 1000 may be connected to a communication network. The server 1000 may be connected to other external devices via the communication network. The server 1000 may transmit data to the other devices connected to the server 1000, or may receive data from the other devices. The communication network connected to the server 1000 may include a wired communication network, a wireless communication network, or a combined communication network. The communication network may include a mobile communication network such as 3G, LTE, or LTE-A. The communication network may include a wired or wireless communication network such as Wi-Fi, UMTS / GPRS, or Ethernet. The communication network may include a local area network such as Magnetic Secure Transmission (MST), Radio Frequency Identification (RFID), Near Field Communication (NFC), ZigBee, Z-Wave, Bluetooth, Bluetooth Low Energy (BLE), or InfraRed communication (IR). The communication network may include a local area network (LAN), a metropolitan area network (MAN), or a wide area network (WAN), etc.
[0034] The server 1000 may receive data from at least one of the first terminal 2000 and the second terminal 3000. The server 1000 may perform a calculation using the data received from at least one of the first terminal 2000 and the second terminal 3000. The server 1000 may transmit a result of the calculation to at least one of the first terminal 2000 and the second terminal 3000. The server 1000 may receive an intermediation request from at least one of the first terminal 2000 and the second terminal 3000. The server 1000 may select a terminal to which the intermediation request is to be transmitted. For example, the server 1000 may select the first terminal 2000 and the second terminal 3000. The server 1000 may mediate a communication connection between the selected first terminal 2000 and second terminal 3000. For example, the server 1000 may mediate a video call connection between the first terminal 2000 and the second terminal 3000, or may mediate a text transmission / reception connection. The server 1000 may transmit connection information related to the first terminal 2000 to the second terminal 3000, or may transmit connection information related to the second terminal 3000 to the first terminal 2000. The connection information regarding the first terminal 2000 may include, for example, an IP address and a port number of the first terminal 2000. The first terminal 2000 that has received the connection information regarding the second terminal 3000 may attempt to connect to the second terminal 3000 using the received connection information.
[0035] A video call session between the first terminal 2000 and the second terminal 3000 may be established by a successful attempt to connect the first terminal 2000 to the second terminal 3000 or a successful attempt to connect the second terminal 3000 to the first terminal 2000. Through the video call session, the first terminal 2000 may transmit images and sounds to the second terminal 3000. The first terminal 2000 may encode the images and sounds into digital signals and transmit the encoded results to the second terminal 3000. The first terminal 2000 may receive images and sounds encoded in a digital signal and decode the received images and sounds. Through the video call session, the second terminal 3000 may transmit images and sounds to the first terminal 2000. Also, through the video call session, the second terminal 3000 may receive images and sounds from the first terminal 2000. In this way, the user of the first terminal 2000 and the user of the second terminal 3000 can make a video call with each other. The first terminal 2000 and the second terminal 3000 may be, for example, a desktop computer, a laptop computer, a smartphone, a smart tablet, a smart watch, a mobile terminal, a digital camera, a wearable device, or a portable electronic device. The first terminal 2000 and the second terminal 3000 may execute a program or an application. The first terminal 2000 and the second terminal 3000 may be the same type of device or may be different types of devices.
[0036] 16 is a flow chart showing a landmark separation method according to an embodiment of the present invention. Referring to FIG. 16, the landmark separation method according to an embodiment of the present invention includes a step of receiving a face image and landmark information (S210), a step of estimating a transformation matrix (S220), a step of calculating expression landmarks (S230), and a step of calculating identity landmarks (S240). In step S210, a facial image of a first person and landmark information corresponding to the facial image are received. Here, the landmark may be understood as a facial landmark of the facial image. The landmark may refer to major elements of a face, such as eyes, eyebrows, nose, mouth, jaw line, etc. The landmark information may also include information regarding the position, size, or shape of the main features of the face, and may also include information regarding the color or texture of the main features of the face.
[0037] The first person means any person, and in step S210, a face image of the any person and landmark information corresponding to the face image are received. The landmark information is obtained using a known technique, and any known method may be used. The method of acquiring the landmarks does not limit the present invention. In step S220, a transformation matrix corresponding to the landmark information is estimated. The transformation matrix may constitute the landmark information together with a predetermined unit vector. For example, the first landmark information may be calculated by multiplying the unit vector by the first transformation matrix. Also, the second landmark information may be calculated by multiplying the unit vector by the second transformation matrix. The transformation matrix is a matrix that transforms high-dimensional landmark information into low-dimensional data, and may be used in Principal Component Analysis (PCA). PCA is a dimensionality reduction method that searches for new mutually orthogonal axes while maximally preserving the variance of the data, and transforms variables in a high-dimensional space into variables in a low-dimensional space. PCA first finds a hyperplane that is closest to the data, and then projects the data onto the low-dimensional hyperplane to reduce the dimension of the data. In PCA, the unit vector that defines the i-th axis is taken as the i-th principal component (PC), and high-dimensional data can be converted to low-dimensional data by linearly combining these axes.
number
[0038] Therefore, in step S220, the face image of the first person and landmark information corresponding to the face image are received as input, and a transformation matrix is estimated and output therefrom. On the other hand, the learning model may be trained to classify landmark information into a plurality of semantic groups corresponding to the right eye, the left eye, the nose, and the mouth, and to output PCA transform coefficients corresponding to each of the plurality of semantic groups. At this time, the semantic groups are not necessarily classified to correspond to the right eye, left eye, nose, and mouth, but may be classified to correspond to eyebrows, eyes, nose, mouth, and jaw line, or may be classified to correspond to eyebrows, right eye, left eye, nose, mouth, jaw line, ears, etc. In step S120, the landmark information may be classified into semantic groups of finer units according to the learning model, and PCA transformation coefficients corresponding to the classified semantic groups may be estimated. In step S230, the expression landmarks of the first person are calculated using the transformation matrix. The landmark information can be decomposed into a plurality of sub landmark information, but in the present invention, the landmark information is represented as follows:
number
[0039] Here, l(c, t) is the landmark information of the t-th frame of the video containing person c, and l m is the mean facial landmark information for a person, l id (c) is the facial landmark of identity geometry information of person c, exp (c, t) denotes the facial landmark of expression geometry of person c in the t-th frame of a video including person c. In other words, the landmark information for a particular person in a particular frame may be represented as the sum of the average landmark information of all people's faces, the identity landmark information of only the particular person, and the facial expression and movement information of the particular person in the particular frame. The average landmark information can be defined by the following formula and may be calculated based on many images that can be collected in advance.
number
number
[0040] In other words, b exp denotes the eigenvectors previously described, and high-dimensional representation landmarks may be defined by combinations of low-dimensional eigenvectors. expmeans the total number of expressions and movements that person c can make using the right eye, left eye, nose, mouth, etc. Therefore, the expression landmarks of the first person may be defined as a set of expression information for each of the main parts of the face, that is, the right eye, the left eye, the nose, and the mouth. k (c, t) may exist corresponding to each eigenvector. The learning model may be trained to estimate the PCA coefficient α(c,t) by inputting a photograph x(c,t) of a person c whose landmark information is to be separated and landmark information l(c,t) as shown in Equation 8. Through such training, the learning model may estimate a PCA coefficient from an image of a specific person and its corresponding landmark information, or may estimate the low-dimensional eigenvector. When applying a trained neural network, the photo x(c', t) of a person c' for which landmark separation is to be performed and the landmark information l(c', t) are input to the neural network, and the PCA transformation matrix is estimated. exp are the predicted (estimated) PCA coefficients using values obtained from the training data, and b exp Using this, we can estimate the representation landmarks as follows:
number
[0041] In step S240, the first person's identity landmarks are calculated using the representation landmarks. As described with reference to Equation 2, the landmark information may be defined as the sum of average landmark information, identity landmark information, and representation landmark information, and the representation landmark information may be estimated in step S230 using Equation 11. Therefore, the identity landmarks can be calculated as follows:
number
[0042] FIG. 17 is a schematic diagram of a method for calculating a transformation matrix according to an embodiment of the present invention. Referring to FIG. 17, an artificial neural network receives an input image of a face of an arbitrary person. The artificial neural network may be a part of a known artificial neural network, but in one embodiment, the artificial neural network may be ResNet. ResNet is a kind of CNN (Convolutional Neural Network), and the present invention is not limited to a specific type of artificial neural network. MLP (Multi-Layer Perceptron) is a kind of artificial neural network in which multiple layers of Perceptrons are stacked to overcome the limitations of single-layer Perceptron. Referring to Fig. 17, MLP receives the output of the artificial neural network and landmark information corresponding to the face image as input. Also, MLP outputs a transformation matrix. In FIG. 17, the artificial neural network and the MLP may be understood as constituting one trained artificial neural network as a whole. Once the transformation matrix is estimated through the trained artificial neural network, the expression landmark information and identity landmark information can be calculated as described with reference to Figure 16. The landmark separation method according to the present invention can also be applied when there are only a very small number of face images or only one frame of face images.
[0043] The trained artificial neural network is trained to estimate low-dimensional eigenvectors and transformation coefficients from a large number of face images and their corresponding landmark information, and the trained artificial neural network is capable of estimating the eigenvectors and transformation coefficients even when only one frame of a face image is given. Separating expression and identity landmarks for any person in this way can improve the quality of facial image processing techniques such as facial landmark-based face reconstruction, face classification, and face morphing. Facial replay technology is a technique that synthesizes facial images and photographs that, given a target face and a driver face, mimic the movements of the driver's face but with the identity of the target face. Face morphing technology is a technology that synthesizes a face image or photo of a third person with the characteristics of person 1 and person 2 when face images or photos of person 1 and person 2 are given. Traditional morphing algorithms find face key points, and then divide the face into non-overlapping triangular or rectangular shapes based on the key points. Then, the photos of person 1 and person 2 are aligned to synthesize a photo of the third person. However, since the positions of the key points of person 1 and person 2 are different from each other, it may feel unnatural when the photos of person 1 and person 2 are aligned pixel-wise to generate a photo of the third person. Known face morphing technology does not distinguish between the target's appearance features and characteristics based on emotions such as facial expressions, so the quality of the morphed result may be low. The landmark separation method according to the present invention can separate expression landmark information and identity landmark information from one piece of landmark information, and therefore can contribute to improving the results of face image processing techniques that utilize face landmarks. In particular, the landmark separation method according to the present invention is very useful because it can separate landmarks even when only a very small amount of face image data is provided.
[0044] 18 is a diagram illustrating a schematic configuration of a landmark separating device according to an embodiment of the present invention. Referring to FIG. 18, a landmark separating device 5000 according to an embodiment of the present invention includes a receiving unit 5100, a transformation matrix estimating unit 5200, and a computing unit 5300. The receiving unit 5100 receives a face image of a first person and landmark information corresponding to the face image. Here, the landmark may be understood as a concept including main elements of a face as the face landmark, such as eyes, eyebrows, nose, mouth, and jaw line. The landmark information may also include information regarding the position, size, or shape of the main features of the face, and may also include information regarding the color or texture of the main features of the face. The first person means any person, and the receiving unit 5100 receives a face image of the any person and landmark information corresponding to the face image. The landmark information is obtained using a known technique, and any known method may be used. The present invention is not limited by the method of acquiring the landmarks. The transformation matrix estimation unit 5200 estimates a transformation matrix corresponding to the landmark information. The transformation matrix may constitute the landmark information together with a predetermined unit vector. For example, the first landmark information may be calculated by multiplying the unit vector and the first transformation matrix. Also, the second landmark information may be calculated by multiplying the unit vector and the second transformation matrix. The transformation matrix is a matrix that transforms high-dimensional landmark information into low-dimensional data, and may be used in Principal Component Analysis (PCA). PCA is a dimensionality reduction method that searches for new mutually orthogonal axes while maximally preserving the variance of the data, and transforms variables in a high-dimensional space into variables in a low-dimensional space. PCA first finds a hyperplane that is closest to the data, and then projects the data onto the low-dimensional hyperplane to reduce the dimension of the data. In PCA, the unit vector that defines the i-th axis is taken as the i-th principal component (PC), and high-dimensional data can be converted to low-dimensional data by linearly combining these axes.
[0045] As described above, the unit vectors, i.e., the principal components, may be determined in advance. Therefore, when new landmark information is received, the corresponding transformation matrix may be determined. At this time, there may be multiple transformation matrices corresponding to one landmark information. On the other hand, the transformation matrix estimation unit 5200 may use a learning model trained to estimate the above transformation matrix. The learning model may be understood as a model trained to estimate a PCA transformation matrix from an arbitrary face image and landmark information corresponding to the arbitrary face image. The learning model may be trained to estimate the transformation matrix from face images of different people and landmark information corresponding to each face image. Although there may be a plurality of transformation matrices corresponding to one piece of high-dimensional landmark information, the learning model may be trained to output only one transformation matrix among the plurality of transformation matrices. The landmark information used as input to the learning model may be obtained using known methods of extracting landmarks from a face image and visualizing them. Therefore, the transformation matrix estimation unit 5200 receives the face image of the first person and landmark information corresponding to the face image as an input, and estimates and outputs one transformation matrix therefrom. On the other hand, the learning model may be trained to classify landmark information into a plurality of semantic groups corresponding to the right eye, the left eye, the nose, and the mouth, and to output PCA transform coefficients corresponding to each of the plurality of semantic groups. In this case, the semantic groups are not necessarily classified to correspond to the right eye, left eye, nose, and mouth, but may be classified to correspond to eyebrows, eyes, nose, mouth, and chin line, or may be classified to correspond to eyebrows, right eye, left eye, nose, mouth, chin line, ears, etc. The transformation matrix estimation unit 5200 may classify the landmark information into semantic groups of finer units according to the learning model, and estimate PCA transformation coefficients corresponding to the classified semantic groups.
[0046] The calculation unit 5300 calculates the representation landmarks of the first person using the transformation matrix, and calculates the identity landmarks of the first person using the representation landmarks. The landmark information may be separated into a plurality of sub-landmark information, for example, into average landmark information, identity landmark information, and representation landmark information. In other words, the landmark information for a particular person in a particular frame may be represented as the sum of the average landmark information of all people's faces, the identity landmark information of only the particular person, and the facial expression and movement information of the particular person in the particular frame. The average landmark information can be defined by the following formula and may be calculated based on many images that can be collected in advance. The learning model may be trained to estimate the PCA coefficient α(c,t) by inputting a photograph x(c,t) of a person c whose landmark information is to be separated and landmark information l(c,t) as shown in Equation 8. Through such training, the learning model may estimate a PCA coefficient from an image of a specific person and its corresponding landmark information, or may estimate the low-dimensional eigenvector. When applying a trained neural network, the photo x(c', t) of a person c' for which landmark separation is to be performed and the landmark information l(c', t) are input to the neural network, and the PCA transformation matrix is estimated. exp are the predicted (estimated) PCA coefficients using values obtained from the training data, and b exp The expression landmarks may be estimated using Equation 11. On the other hand, as described with reference to Equation 8, the landmark information may be defined as the sum of the average landmark information, the identity landmark information, and the representation landmark information, and the representation landmark information may be estimated using Equation 11 in step S230. Therefore, the identity landmarks may be calculated as shown in Equation 12. Given a face image of any person, landmark information may be obtained from it, and expression landmark information and identity landmark information may be calculated from the face image and landmark information.
[0047] Fig. 19 is a diagram showing an example of a method for recreating a face using the present invention. Referring to Fig. 19, a target image 4100 and a driver image 4200 are shown, and the target image 4100 may replay an image corresponding to the driver image 4200. It can be seen that the re-enacted image 4300 has the characteristics of the target image 4100, but its facial expression corresponds to that of the driver image 4200. That is, the re-enacted image 4300 has the identity landmarks of the target image 4100, but the expression landmarks have characteristics corresponding to those of the driver image 4200. Therefore, it turns out that for natural face reconstruction, it is important to properly separate identity and expression landmarks from a single landmark.
[0048] Fig. 20 is a diagram illustrating an environment in which an image transformation device and an image transformation method according to the present invention operate. Referring to Fig. 20, the environment in which a first terminal 6000 and a second terminal 7000 operate may include a server 10000, and a first terminal 6000 and a second terminal 7000 connected to the server 10000. For convenience of explanation, Fig. 20 shows only two terminals, i.e., the first terminal 6000 and the second terminal 7000, but two or more terminals may be included. For terminals that may be added, the explanations regarding the first terminal 6000 and the second terminal 7000 may be applied, except for the explanations to be specifically mentioned. The server 10000 may be connected to a communication network. The server 10000 may be connected to other external devices via the communication network. The server 10000 may transmit data to the other devices connected to each other, or may receive data from the other devices. The communication network connected to the server 10000 may include a wired communication network, a wireless communication network, or a combined communication network. The communication network may include a mobile communication network such as 3G, LTE, or LTE-A. The communication network may include a wired or wireless communication network such as Wi-Fi, UMTS / GPRS, or Ethernet. The communication network may include a local area network such as Magnetic Secure Transmission (MST), Radio Frequency Identification (RFID), Near Field Communication (NFC), ZigBee, Z-Wave, Bluetooth, Bluetooth Low Energy (BLE), or InfraRed communication (IR). The communication network may include a local area network (LAN), a metropolitan area network (MAN), or a wide area network (WAN), etc.
[0049] The server 10000 may receive data from at least one of the first terminal 6000 and the second terminal 7000. The server 10000 may perform a calculation using the data received from at least one of the first terminal 6000 and the second terminal 7000. The server 10000 may transmit a result of the calculation to at least one of the first terminal 6000 and the second terminal 7000. The server 10000 may receive an intermediation request from at least one of the first terminal 6000 and the second terminal 7000. The server 10000 may select a terminal to which the intermediation request is to be transmitted. For example, the server 10000 may select the first terminal 6000 and the second terminal 7000. The server 10000 may mediate a communication connection between the selected first terminal 6000 and second terminal 7000. For example, the server 10000 may mediate a video call connection between the first terminal 6000 and the second terminal 7000, or may mediate a text transmission / reception connection. The server 10000 may transmit connection information related to the first terminal 6000 to the second terminal 7000, or may transmit connection information related to the second terminal 7000 to the first terminal 6000. The connection information regarding the first terminal 6000 may include, for example, an IP address and a port number of the first terminal 6000. The first terminal 6000 that has received the connection information regarding the second terminal 7000 may attempt to connect to the second terminal 7000 using the received connection information.
[0050] A video call session between the first terminal 6000 and the second terminal 7000 may be established by a successful attempt to connect the first terminal 6000 to the second terminal 7000 or a successful attempt to connect the second terminal 7000 to the first terminal 6000. Through the video call session, the first terminal 6000 may transmit images and sounds to the second terminal 7000. The first terminal 6000 may encode the images and sounds into digital signals and transmit the encoded results to the second terminal 7000. Furthermore, through the video call session, the first terminal 6000 may receive images and sounds from the second terminal 7000. The first terminal 6000 may receive images and sounds encoded in digital signals and decode the received images and sounds. Through the video call session, the second terminal 7000 may transmit images and sounds to the first terminal 6000. Also, through the video call session, the second terminal 7000 may receive images and sounds from the first terminal 6000. In this way, the user of the first terminal 6000 and the user of the second terminal 7000 can make a video call with each other. The first terminal 6000 and the second terminal 7000 may be, for example, a desktop computer, a laptop computer, a smartphone, a smart tablet, a smart watch, a mobile terminal, a digital camera, a wearable device, or a portable electronic device. The first terminal 6000 and the second terminal 7000 may execute a program or an application. The first terminal 6000 and the second terminal 7000 may be the same type of device or may be different types of devices.
[0051] FIG. 21 is a flow chart that illustrates an image deformation method according to an embodiment of the present invention. Referring to FIG. 21, the image transformation method according to one embodiment of the present invention includes a step of acquiring a user's facial landmark information (S2100), a step of generating a user feature map (S2200), a step of generating a target feature map (S2300), a step of generating a mixed feature map (S2400), and a step of generating a reenacted image (S2500). In step S2100, landmark information is obtained from a face image of a user. The landmark means a facial feature of the user, and may include, for example, the user's eyes, eyebrows, nose, mouth, ears, or jaw line. The landmark information may also include information regarding the position, size, or shape of a major element of the user's face. Furthermore, the landmark information may also include information regarding the color or texture of a major element of the user's face. The user may refer to any user who uses a terminal on which the image transformation method according to the present invention is executed. In step S2100, a face image of the user is received, and landmark information corresponding to the face image is acquired. The landmark information is acquired using a known technique, and any known method may be used. The method of acquiring the landmark information does not limit the present invention. In step S2200, a transformation matrix corresponding to the landmark information may be estimated. The transformation matrix may constitute the landmark information together with a predetermined unit vector. For example, the first landmark information may be calculated by multiplying the unit vector by the first transformation matrix. Also, the second landmark information may be calculated by multiplying the unit vector by the second transformation matrix.
[0052] The transformation matrix is a matrix that transforms high-dimensional landmark information into low-dimensional data, and may be used in Principal Component Analysis (PCA). PCA is a dimensionality reduction method that searches for new mutually orthogonal axes while maximally preserving the variance of the data, and transforms variables in a high-dimensional space into variables in a low-dimensional space. PCA first finds a hyperplane that is closest to the data, and then projects the data onto the low-dimensional hyperplane to reduce the dimension of the data. In PCA, the unit vector that defines the i-th axis is taken as the i-th principal component (PC), and high-dimensional data can be converted to low-dimensional data by linearly combining these axes.
number
[0053] The learning model may be trained to estimate the transformation matrix from face images of different people and landmark information corresponding to each face image. Although there may be a plurality of transformation matrices corresponding to one piece of high-dimensional landmark information, the learning model may be trained to output only one transformation matrix among the plurality of transformation matrices. The landmark information used as input to the learning model may be obtained using known methods of extracting landmarks from a face image and visualizing them. Therefore, in step S2100, the face image of the user and landmark information corresponding to the face image are received as input, and a transformation matrix is estimated and output therefrom. On the other hand, the learning model may be trained to classify landmark information into a plurality of semantic groups corresponding to the right eye, the left eye, the nose, and the mouth, and to output PCA transform coefficients corresponding to each of the plurality of semantic groups. In this case, the semantic groups are not necessarily classified to correspond to the right eye, left eye, nose, and mouth, but may be classified to correspond to eyebrows, eyes, nose, mouth, and jaw line, or may be classified to correspond to eyebrows, right eye, left eye, nose, mouth, jaw line, ears, etc. In step S2100, the landmark information may be classified into semantic groups of finer units according to the learning model, and PCA transformation coefficients corresponding to the classified semantic groups may be estimated.
[0054] Meanwhile, the user's expression landmarks are calculated using the transformation matrix. The landmark information can be decomposed into a plurality of sub-landmark information, but in the present invention, the landmark information is represented as follows:
number
number
number
[0055] The learning model may be trained to estimate the PCA coefficient α(c,t) by inputting a photograph x(c,t) of a person c whose landmark information is to be separated and landmark information l(c,t) as shown in Equation 14. By such training, the learning model may estimate a PCA coefficient from an image of a specific person and its corresponding landmark information, or may estimate the low-dimensional eigenvector. When applying a trained neural network, the photo x(c', t) of a person c' for which landmark separation is to be performed and the landmark information l(c', t) are input to the neural network, and the PCA transformation matrix is estimated. exp are the predicted (estimated) PCA coefficients using values obtained from the training data, and b exp Using this, we can estimate the representation landmarks as follows:
number
number
[0056] The user feature map generated in step S2200 includes information representing the features of the facial expressions and facial movements of the user. The artificial neural network used in step S2200 may be a Convolutional Neural Network (CNN), but various types of artificial neural networks may also be used. In step S2300, a target face image is received, and a target feature map and a pose-normalized target feature map are generated based on style information and pose information corresponding to the target face image. The target refers to a person to be transformed by the present invention, and the user and the target may be different people, but are not necessarily limited to this. A reenacted image generated as a result of the implementation of the present invention may be displayed as the target's appearance that is transformed from the facial image of the target and mimics or copies the movements and expressions of the user. The target feature map includes information that represents the facial expressions exhibited by the target and the characteristics of the facial movements of the target. The pose-normalized target feature map may correspond to an output of an artificial neural network for the input style information, or the pose-normalized target feature map may include information corresponding to distinctive features of the target's face, exclusive of the target's pose information.
[0057] The artificial neural network used in step S2300 may be a CNN, similar to the artificial neural network used in step S2200, and the structure of the artificial neural network used in step S2200 may be different from the structure of the artificial neural network used in step S2300. The style information means information on a person's face that indicates the unique features of that person, and may include, for example, innate features appearing on the target's face, the size, shape, and position of landmarks, etc. Alternatively, the style information may include at least one of texture information, color information, and shape information corresponding to the target's facial image.
[0058] The target feature map may be understood as including data corresponding to expression landmark information obtained from a facial image of the target, and the pose-normalized target feature map may be understood as including data corresponding to identity landmark information obtained from a facial image of the target. Alternatively, the style information may include at least one of texture information, color information, and shape information corresponding to the facial image of the target. The mixed feature map may be generated such that the target landmarks have pose information that corresponds to the user's landmarks. The artificial neural network used in step S2400 may be a CNN, similar to the artificial neural networks used in steps S2200 and S2300, and the structure of the artificial neural network used in step S2400 may be different from the structures of the artificial neural networks used in the previous steps.
[0059] In step S2500, the mixed feature map and the pose-normalized target feature map are used to generate a reconstructed image for the target face image. As described above, the pose-normalized target feature map includes data corresponding to identity landmark information obtained from the facial image of the target, where the identity landmark information refers to information corresponding to unique features of a person that is unrelated to expression information corresponding to the person's movement information or facial expression information. If a target's movement can be obtained that naturally follows the user's movement through the mixed feature map generated in step S2400, then in step S2500, the unique characteristics of the target can be reflected, resulting in an effect similar to that of an actual target moving and expressing facial expressions on its own.
[0060] 22 is a diagram showing an example of a result of performing an image transformation method according to an embodiment of the present invention, showing a target image, a user image, and a reenacted image, where the reenacted image has the facial movements and expressions of the user while preserving the facial features of the target. Comparing the target image and the reenacted image in Figure 22, we can see that the two images show the same person, with only differences in facial expressions. The eyes, nose, mouth, and hairstyle of the target image are similar to those of the reenacted image, respectively. Meanwhile, the facial expression of the person in the replayed image is substantially similar to the facial expression of the user, for example, if the user has his / her mouth open in the image of the user, the replayed image will have the image of the target with his / her mouth open, and if the user has his / her head turned right or left in the image of the user, the replayed image will have the image of the target with his / her head turned right or left. When a real-time changing image of a user is received and a replayed image is generated based thereon, the replayed image can modify the target image in response to the real-time changing movements and facial expressions of the user.
[0061] 23 is a diagram illustrating an outline of the configuration of an image transformation device according to an embodiment of the present invention. Referring to FIG. 23, an image transformation device 8000 according to an embodiment of the present invention includes a landmark acquisition unit 8100, a first encoder 8200, a second encoder 8300, a blender 8400, and a decoder 8500. The landmark acquisition unit 8100 receives face images of a user and a target, and acquires landmark information from each face image. The landmark means a facial part that is a characteristic of the user's face, and may include, for example, the user's eyes, eyebrows, nose, mouth, ears, or jaw line. The landmark information may also include information regarding the position, size, or shape of a major element of the user's face. Furthermore, the landmark information may also include information regarding the color or texture of a major element of the user's face. The user may refer to any user who uses a terminal on which the image transformation method according to the present invention is executed. The landmark acquisition unit 8100 receives a face image of the user and acquires landmark information corresponding to the face image. The landmark information is acquired using a known technique, and any known method may be used. The present invention is not limited by the method of acquiring the landmark information.
[0062] The landmark acquisition unit 8100 may estimate a transformation matrix corresponding to the landmark information. The transformation matrix may constitute the landmark information together with a predetermined unit vector. For example, the first landmark information may be calculated by multiplying the unit vector and the first transformation matrix. Also, the second landmark information may be calculated by multiplying the unit vector and the second transformation matrix. The transformation matrix is a matrix that transforms high-dimensional landmark information into low-dimensional data, and may be used in Principal Component Analysis (PCA). PCA is a dimensionality reduction method that searches for new mutually orthogonal axes while maximally preserving the variance of the data, and transforms variables in a high-dimensional space into variables in a low-dimensional space. PCA first finds a hyperplane that is closest to the data, and then projects the data onto the low-dimensional hyperplane to reduce the dimension of the data. In PCA, the unit vector that defines the i-th axis is taken as the i-th principal component (PC), and high-dimensional data can be converted to low-dimensional data by linearly combining these axes.
[0063] On the other hand, the landmark acquisition unit 8100 may use a learning model trained to estimate the above transformation matrix. The above learning model may be understood as a model trained to estimate a PCA transformation matrix from an arbitrary face image and landmark information corresponding to the arbitrary face image. The learning model may be trained to estimate the transformation matrix from face images of different people and landmark information corresponding to each face image. Although there may be a plurality of transformation matrices corresponding to one piece of high-dimensional landmark information, the learning model may be trained to output only one transformation matrix among the plurality of transformation matrices. The landmark information used as input to the learning model may be obtained using known methods of extracting landmarks from a face image and visualizing them. Therefore, the landmark acquisition unit 8100 receives the face image of the user and landmark information corresponding to the face image as input, and estimates and outputs one transformation matrix therefrom. On the other hand, the learning model may be trained to classify landmark information into a plurality of semantic groups corresponding to the right eye, the left eye, the nose, and the mouth, and to output PCA transform coefficients corresponding to each of the plurality of semantic groups. In this case, the semantic groups are not necessarily classified to correspond to the right eye, left eye, nose, and mouth, but may be classified to correspond to eyebrows, eyes, nose, mouth, and chin line, or may be classified to correspond to eyebrows, right eye, left eye, nose, mouth, chin line, ears, etc. The landmark acquiring unit 8100 may classify the landmark information into semantic groups of subdivided units according to the learning model, and estimate PCA transformation coefficients corresponding to the classified semantic groups.
[0064] Meanwhile, the transformation matrix may be used to calculate the expression landmarks of the user. The landmark information may be separated into a plurality of sub-landmark information, but in the present invention, the landmark information is defined as the sum of the average face landmark information of a person, the individual specific face landmark information of the person, and the expression face landmark information of the person. In other words, the landmark information for a particular person in a particular frame may be represented as the sum of the average landmark information of all people's faces, the identity landmark information of only the particular person, and the facial expression and movement information of the particular person in the particular frame. Meanwhile, the expression landmarks correspond to pose information of the user's facial image, and the identity landmarks correspond to style information of the target's facial image. In summary, the landmark acquisition unit 8100 may receive a facial image of the user and a facial image of the target, and generate multiple landmark information therefrom, including respective expression landmark information and identity landmark information. The first encoder 8200 generates a user feature map from pose information of the user's facial image. The pose information corresponds to the expression landmark information and may include movement information and facial expression information of the facial image. The first encoder 8200 may also input the pose information corresponding to the user's facial image into an artificial neural network to generate the user feature map.
[0065] The user feature map generated by the first encoder 8200 includes information representing the features of the facial expressions and facial movements of the user. The artificial neural network used in the first encoder 8200 may be a Convolutional Neural Network (CNN), but various types of artificial neural networks may also be used. The second encoder 8300 generates a target feature map and a pose-normalized target feature map from the style information and pose information of the target face image. The target refers to a person to be transformed by the present invention, and the user and the target may be different people, but are not necessarily limited to this. A reenacted image generated as a result of the implementation of the present invention may be displayed as the target's appearance that is transformed from the facial image of the target and mimics or copies the movements and expressions of the user. The target feature map generated by the second encoder 8300 may be understood as data corresponding to the user feature map generated by the first encoder 8200 and includes information representing the facial expressions exhibited by the target and the characteristics of the facial movements of the target. The pose-normalized target feature map may correspond to an output of an artificial neural network for the input style information, or the pose-normalized target feature map may include information corresponding to distinctive features of the target's face, exclusive of the target's pose information.
[0066] The artificial neural network used in the second encoder 8300 may be a CNN, similar to the artificial neural network used in the first encoder 8200, and the structure of the artificial neural network used in the first encoder 8200 and the structure of the artificial neural network used in the second encoder 8300 may be different from each other. The style information means information on a person's face that indicates the unique features of that person, and may include, for example, innate features appearing on the target's face, the size, shape, and position of landmarks, etc. Alternatively, the style information may include at least one of texture information, color information, and shape information corresponding to the target's facial image. The target feature map may be understood as including data corresponding to expression landmark information obtained from a facial image of the target, and the pose-normalized target feature map may be understood as including data corresponding to identity landmark information obtained from a facial image of the target. Blender 8400 may generate a mixed feature map using the user feature map and the target feature map, and input pose information of the user's facial image and style information of the target's facial image into an artificial neural network to generate the mixed feature map.
[0067] The mixed feature map may be generated such that the target's landmarks have pose information corresponding to the user's landmarks. The artificial neural network used in Blender 8400 may be a CNN similar to the artificial neural networks used in the first encoder 8200 and the second encoder 8300, and the structure of the artificial neural network used in Blender 8400 may be different from the structure of the artificial neural network used in the first encoder 8200 or the second encoder 8300. The user feature map and the target feature map input to the blender 8400 contain user facial landmark information and target facial landmark information, respectively, to generate a target face corresponding to the facial movements and expressions of the user, and may perform an operation of matching the user's facial landmarks with the target facial landmarks so as to maintain the unique features of the target face. For example, in order to control the facial movement of the target in line with the facial movement of the user, landmarks such as the user's eyes, eyebrows, nose, mouth, jaw line, etc. may be understood as being linked to landmarks such as the eyes, eyebrows, nose, mouth, jaw line, etc. of the target, respectively.
[0068] Alternatively, the user's landmarks such as the eyes, eyebrows, nose, mouth, jaw line, etc. may be linked to the target's landmarks such as the eyes, eyebrows, nose, mouth, jaw line, etc., in order to control the target's facial expressions in line with the user's facial expressions. The decoder 8500 uses the mixed feature map and the pose-normalized target feature map to generate a reconstructed image for the target face image. As described above, the pose-normalized target feature map includes data corresponding to identity landmark information obtained from the facial image of the target, where the identity landmark information refers to information corresponding to unique features of a person that is unrelated to expression information corresponding to the person's movement information or facial expression information. If the target's movements can be obtained to naturally follow the user's movements via the mixed feature map generated by Blender 8400, the unique characteristics of the target can be reflected in the decoder 8500, resulting in an effect similar to that of a real target moving and expressing facial expressions on its own.
[0069] FIG. 24 is a diagram illustrating a schematic configuration of a landmark acquisition unit according to an embodiment of the present invention. Referring to FIG. 24, the landmark acquisition unit according to an embodiment of the present invention may include an artificial neural network, which receives a face image of a person as an input. The artificial neural network may be a part of a known artificial neural network, but in one embodiment, the artificial neural network may be ResNet. ResNet is a type of CNN (Convolutional Neural Network), and the present invention is not limited to a specific type of artificial neural network. MLP (Multi-Layer Perceptron) is a kind of artificial neural network in which multiple layers of Perceptrons are stacked to overcome the limitations of single-layer Perceptron. Referring to FIG. 24, the MLP receives the output of the artificial neural network and landmark information corresponding to the face image as input. The MLP also outputs a transformation matrix. Meanwhile, the artificial neural network and the MLP may be understood as constituting one trained artificial neural network as a whole. Once the transformation matrix is estimated through the trained artificial neural network, the expression landmark information and identity landmark information can be calculated as described with reference to Fig. 23. The image transformation device according to the present invention can also be applied to cases where only a very small number of face images are present or where only one frame of face images are present. The trained artificial neural network is trained to estimate low-dimensional eigenvectors and transformation coefficients from a large number of face images and their corresponding landmark information, and the trained artificial neural network is capable of estimating the eigenvectors and transformation coefficients even when only one frame of a face image is given. In this way, separation of expression and identity landmarks for any person can be used to improve the quality of facial image processing techniques such as facial landmark-based face reenactment, face classification, and face morphing.
[0070] FIG. 25 is a diagram illustrating a schematic configuration of a second encoder according to an embodiment of the present invention. 25, the second encoder 8300 according to an embodiment of the present invention may adopt a U-Net structure. U-Net means a U-shaped network, which basically performs a subdivision function and has a symmetrical form. f y denotes the normalized flow map used to normalize the target feature map, and T denotes the warping function that performs the warping. Also, S j、 j=1....n y denotes the target feature map encoded in each convolutional layer. The second encoder 8300 receives the rendered target landmarks and the target image as input, and then generates an encoded target feature map and a regularized flow map f y Also, the generated target feature map S jand the normalized flow map f y We take,x,=,y,=,y,=,y,and perform a warping function to generate a warped target feature map. The warped target feature map here may be understood as being similar to the pose-normalized target feature map described above. Thus, the warping function T may be understood as a function that generates data consisting of only the style information of the target, i.e., only the identity landmark information, excluding the expression landmark information of the target. FIG. 26 is a schematic diagram showing the structure of a blender according to one embodiment of the present invention. As described above, the Blender 8400 generates a mixed feature map from a user feature map and a target feature map, but the pose information of the user's facial image and the style information of the target's facial image may also be input to an artificial neural network to generate the mixed feature map.
[0071] Although Fig. 26 shows one user feature map and three target feature maps, the number of target feature maps may be one, two, or more than three. Also, the small area inside each feature map shown in Fig. 25 represents information for an arbitrary landmark, and all of them show information for the same landmark. The user feature map and the target feature map input to the blender 8400 contain user facial landmark information and target facial landmark information, respectively, to generate a target face corresponding to the facial movements and expressions of the user, and may perform an operation of matching the user's facial landmarks with the target facial landmarks so as to maintain the unique features of the target face. For example, in order to control the facial movement of the target in line with the facial movement of the user, landmarks such as the user's eyes, eyebrows, nose, mouth, jaw line, etc. may be understood as being linked to landmarks such as the eyes, eyebrows, nose, mouth, jaw line, etc. of the target, respectively. Alternatively, the user's landmarks such as the eyes, eyebrows, nose, mouth, jaw line, etc. may be linked to the target's landmarks such as the eyes, eyebrows, nose, mouth, jaw line, etc., in order to control the target's facial expressions in line with the user's facial expressions. Also, for example, the user feature map may be searched for eyes, and then the target feature map may be searched for eyes, and a mixed feature map may be generated such that the eyes in the target feature map follow the movements of the eyes in the user feature map. Substantially the same operations may be performed by the blender 8400 for other landmarks.
[0072] FIG. 27 is a diagram illustrating the structure of a decoder according to one embodiment of the present invention. Referring to FIG. 27, a decoder 8500 according to one embodiment of the present invention uses the pose-normalized target feature map generated by the second encoder 8300 and the mixed feature map z generated by the blender 8400. xy By taking the input, the user's expression landmark information is applied to the target image. In FIG. 27, the data input to each block of the decoder 8500 is the pose-normalized target feature map generated by the second encoder 8300, f u means a flow map that applies user expression landmark information to the pose-normalized target feature map. In addition, the Warp-alignment block of the decoder 8500 performs a warping function using the output u of the previous block of the decoder 8500 and the pose-normalized target feature map as input. The warping function performed by the decoder 8500 is different from the warping function performed by the second encoder 8300 in that it is for generating a reenacted image that mimics the user's movement and pose while preserving the unique features of the target. Meanwhile, a moving image can be generated based on the embodiments described above with reference to Figures 1 to 27. For example, as described above with reference to Figures 5a to 6b, a moving image can be generated by converting an input still image. Alternatively, an input still image may be converted into a moving image based on an image conversion template. The image conversion template may include a plurality of frames, and each frame may be a still image. For example, each of the plurality of frames may be applied to the input still image to generate a plurality of intermediate images (i.e., a plurality of still images). In addition, the generated intermediate images may be combined to generate a moving image.
[0073] Alternatively, the input video may be converted to generate the video, in which case each of a plurality of first still images (frames) included in the input video is converted into a second still image, and the second still images are combined to generate the video. The embodiments described above with reference to Figs. 1 to 27 may be realized by the contents described below. For example, at least a part of the contents described below may be applied to at least one of the embodiments described above with reference to Figs. 1 to 27. In addition, when the meaning of a term described below is the same as or similar to the meaning of a term described with reference to Figs. 1 to 27, the terms may be understood as terms indicating the same member. Furthermore, when the contents described below are the same as or similar to the contents described above with reference to Figs. 1 to 27, the contents may be understood as being the same. Furthermore, the contents described below may be included in the contents of the paper "MarioNETte: Few-Shot Face Reenactment Preserving Identity of Unseen Targets". When mismatch occurs between the target identity and the driver identity, the quality of the results drops significantly when recreating the face, especially when the imaging is performed over several times. Identity preservation issues, i.e., the model losing target details leading to output defects, are the most common failure modes. This issue has several potential causes, including the leakage of the driver's identity due to identity mismatch and unseen large pose processing.
[0074] To overcome these issues, we propose an image attention block, a target feature alignment block, and a landmark transformation block as components that solve the above-mentioned problems. By processing and warping relevant features, the proposed architecture, called MarioNETte, achieves high-quality reconstruction of identities unseen in a multi-imaging setting. The landmark transformation block also decomposes landmarks to isolate the representation geometry, dramatically mitigating the identity preservation problem. Comprehensive experiments are conducted to verify whether the proposed framework can surpass all other criteria and generate highly realistic faces even when the facial features between the target and the driver are significantly inconsistent. Given a target face and a driver face, facial replay aims to synthesize a face animated by the driver's movements while preserving the target's identity. Traditionally, generative adversarial network (GAN) methods have been widely used, which have achieved great success in image generation tasks. Xu et al.; Wu et al. (2017; 2018) have used CycleGAN (Zhu et al., 2017) to obtain high-fidelity face re-enactment results. However, CycleGAN-based approaches require at least several minutes of training data for each target and can only re-enact predefined identities. This is not very attractive in the real world where re-enactment of unseen targets is unavoidable. Thus, multi-image face reconstruction approaches attempt to reconstruct unseen targets using adaptive instance normalization (AdaIN) (Zakharov et al., 2019) or warping modules (Wiles, Koepke, and Zisserman 2018; Siarohin et al., 2019). However, current state-of-the-art methods suffer from what is called the identity preservation problem, i.e., the inability to preserve the identity of the target, which makes the reconstruction flawed. The problem is exacerbated as the identity of the driver and the identity of the target are different.
[0075] Examples of faulty and successful face reconstructions generated by previous approaches and the proposed model are shown in Figures 28A-28C, respectively. In most cases, the failure of previous approaches can be divided into three modes: 1. Without considering identity mismatch, the driver’s identity inhibits face synthesis, and the generated face resembles the driver (Figure 28A). 2. If the compressed vector representation (e.g., AdaIN layer) is insufficient to preserve the information of the target identity, the generated face may lack detailed features (Figure 28B). 3. The warping process introduces imperfections when dealing with large poses (Figure 28C). We propose a framework called MarioNETte, which aims to recreate the face of an unseen target across several imaging methods while preserving its identity without fine-tuning. We use an image attention block and target feature alignment, which allows MarioNETte to directly inject features from the target when generating images. We also propose a novel landmark transformation unit, which reconciles identity inconsistencies in an unsupervised manner and further mitigates the identity preservation problem. We explain in detail below. We propose a multi-image face replay framework called MarioNETte, which preserves the identity of the target even when the driver's facial features differ significantly from those of the target. The proposed method uses image interest blocks combined with target feature alignment, which involves multiple feature-level warping operations, to allow the model to process the relevant positions of the target feature map, improving the quality of face replay under different identities. We present a new landmark transformation method that accommodates different facial features of different people. The proposed method adapts the driver's landmarks to the target landmarks in an unsupervised manner, mitigating the identity preservation problem without separate label data. We proceed by comparing state-of-the-art methods using the VoxCeleb1 (Nagrani, Chung, and Zisserman 2017) and CelebV (Wu et al. 2018) datasets when the target identity and the driver identity are consistent and distinct, respectively. The experiments, including user studies, show that the proposed method outperforms state-of-the-art methods. MarioNETte structure
[0076] Figure 29 shows the overall structure of the proposed model. The conditional generator G is a set of drivers x and target images We generate a reconstructed face based on JPEG0007579674000038.jpg934, and a classifier D predicts whether the image is real or not. The generator is composed of the following components: The preprocessor P extracts facial keypoints using a 3D landmark detector (Bulat and Tzimiropoulos 2017) and renders landmark images to correspond to the driver and target inputs, respectively. JPEG0007579674000039.jpg626 and JPEG0007579674000040.jpg827 is computed. The proposed landmark transformer is included in the pre-processor. We normalize the size, translation, and rotation of landmarks before using them in the landmark transformer, so we leverage 3D landmarks instead of 2D landmarks. Driver Encoder JPEG0007579674000041.jpg818 extracts pose and expression information from the driver input and creates a feature map of the driver z x Generate. Target Encoder JPEG0007579674000042.jpg723 adopts the U-Net structure to extract style information from the target input and warped target feature map JPEG0007579674000043.jpg85 along with the target feature map z y Generate. Brenda JPEG0007579674000044.jpg943 is the driver feature map z x and the target feature map JPEG0007579674000045.jpg844 is received and mixed feature map z xyThe proposed image attention block is the basic building block of Blender. ·decoder JPEG0007579674000046.jpg845 is the warped target feature map JPEG0007579674000047.jpg75 and mixed feature map z xy The decoder synthesizes the replayed image using the proposed target feature alignment. Image Attention Block To transfer the style information of targets to the driver, previous studies encoded the target information into a vector, which was then concatenated or mixed with the driver features via an AdaIN layer (Liu et al., 2019; Zakharov et al., 2019). However, encoding the target with a spatially agnostic vector results in the loss of the spatial information of the target. In addition, such methods lack a unique design for multiple target images, so they use summary statistics (e.g., average or maximum) to handle multiple targets, which may lose detailed information about the targets.
[0077] To solve the aforementioned problems, we propose an image attention block (Figure 30). The proposed attention block is inspired by the encoder-decoder attention in transforms (VaSwani et al. 2017), where a driver feature map plays the role of attention query and a target feature map plays the role of attention memory. The proposed attention block uses multiple target feature maps (i.e., Z y ), the appropriate location of each feature (red box in Figure 30) is processed. Driver Feature Map JPEG0007579674000048.jpg638 and target feature map Considering JPEG0007579674000049.jpg782, the above attention is calculated as follows:
number
number
[0078] Target Feature Alignment Fine details of target identity can be preserved through warping of low-level features (Siarohin et al. 2019). Unlike previous approaches that calculate the difference between keypoints of the target and the driver and estimate a warping flow map or affine transformation matrix (Balakrishnan et al. 2018; Siarohin et al. 2018; Siarohin et al. 2019), the proposed target feature alignment warps the target feature map in two steps: (1) target pose normalization produces a pose-normalized target feature map, and (2) driver pose adaptation aligns the fully qualified target feature map to the driver's pose (Figure 31). Through the two-step process, the model can better handle the structural differences of different identities. The details are as follows: 1. Target Pose Normalization Target Encoder E y The feature map encoded in JPEG0007579674000055.jpg829 is the estimated normalized flow map f y and the warping function JPEG0007579674000056.jpg876 (1 in Figure 31) The next warp alignment block in the decoder is a target pose agnostic. Process JPEG0007579674000058.jpg96. 2. Driver Pose Adaptation The warp alignment block in the decoder JPEG0007579674000059.jpg1135 and the output u of the previous block of the decoder. In some imaging settings, other target images (e.g., We average the resolution-compatible feature maps of JPEG0007579674000060.jpg1148. To apply the pose-normalized feature maps to the driver pose, we use a 1×1 convolution using u as input and the driver f u Generate an estimated flow map of Alignment is performed with JPEG0007579674000061.jpg1130 (2 in FIG. 31). After that, the above result is concatenated with u and input to the next remaining upsampling block.
[0079] Landmark conversion section Large structural differences between two facial landmarks significantly degrade the quality of the replay. Common approaches to these problems are to learn a transformation for every identity (Wu et al., 2018) or to prepare paired landmark data with similar representations (Zhang et al., 2019). However, such methods are unnatural in a multi-image setting that deals with unseen identities, and there are difficulties in collecting labeled data. To overcome these difficulties, we propose a novel landmark transformer that transfers the driver's facial expression to an arbitrary target identity. The landmark transformer is trained in an unsupervised manner, leveraging multiple images of unlabeled human faces. Landmark Separation When viewing a screen with images of different identities, let x(c, t) be the t-th frame of the c-th image, and let l(c, t) be the 3D facial landmarks. First, we normalize the size, translation, and rotation of all landmarks, and then we define the normalized landmarks as Convert to JPEG0007579674000062.jpg1019. Inspired by 3D morphable face models (Blanz and Vetter 1999), we consider that normalized landmarks can be separated as follows:
number
number
[0080] Landmark Decomposition We introduce a neural network that regresses linear-based coefficients to separate identity and expression geometry in a multi-image setting. Previously, such approaches have been widely used to model complex facial geometry (Blanz and Better 1999). We extract the expression bases from the training data by separating expression landmarks into semantic groups of the face (e.g., mouth, nose, eyes) and performing PCA on each group.
number
number
number
[0081] Experimental setup Dataset We trained our model and baseline using VoxCeleb1 (Nagrani, Chung, and Zisserman 2017), which contains 256 × 256 videos of 1,251 distinct identities. We used test splits of VoxCeleb1 and CelebV (Wu et al. 2018) to evaluate self-repeat and replay under other identities, respectively. We generated a test set by sampling a set of 2,083 images from 100 randomly selected videos from the VoxCeleb1 test split, and uniformly sampled a set of 2,000 images from all identities in CelebV. The CelebV data contains videos of five celebrities with various characteristics, and we use it to evaluate the performance of our model in replaying targets that do not look similar to real-world scenarios. Details of the loss function and training method can be found in Supplementary Materials A3 and A4. standard The MarioNETte transformations with and without the landmark transform (MarioNETte+LT and MarioNETte) are compared to the state-of-the-art models for imaged face reconstruction several times. The details of each criterion are as follows: x2 Faces (Wiles, Koepke, and Zisserman 2018) x2 faces use direct image distortion. We use a pre-trained model provided by the inventors, trained on VoxCeleb1. Monkey-Net (Siarohin et al. 2019) Monkey-Net employs feature-level warping. The implementation provided by the inventors is used. Due to the structure of the method, Monkey-Net can only receive one original image. NeuralHead (Zakharov et al. 2019) NeuralHead leverages AdaIN layers. Since there is no reference implementation, we have made an honest attempt to replicate the results. Our implementation is a feed-forward version of the model (NeuralHead-FF), so we omit the meta-learning and fine-tuning stages. This is because we use a single model to handle multiple identities.
[0082] index To evaluate the quality of the generated images, we compare models based on the following metrics: Structural Similarity (SSIM) (Wang et al. 2004) and Peak Signal-to-Noise Ratio (PSNR), which evaluates the low-level similarity between the generated and real images. We also report Masked SSIM (M-SSIM) and Masked PSNR (M-PSNR), where the measurements are restricted to face regions. In the absence of real images with distinct identities driving the target face, the following metrics are more relevant: We evaluate the identity preservation quality using the cosine similarity (CSIM) of the embedding vectors generated by a pre-trained face recognition model (Deng et al. 2019). To check the model's ability to properly reproduce the pose and expression of the driver, we calculate the PRMSE, which is the root mean square error of the head pose angle, and the PRSE, which is the ratio of similar facial action unit values between the generated image and the driving image. We use OpenFace (Baltrusaitis et al. 2018) to calculate the pose angle and action unit values. Experimental Results We compared the models under self-replay and self-replay of different identities including user studies. Ablation experiments were also performed. All experiments were performed in both single and multiple capture settings, where one target image was used in the single capture case and eight target images were used in the multiple capture case. self-reenactment Figure 34 shows the model evaluation results under the self-replay setting for VoxCeleb1. MarioNETte outperforms other models in all metrics in the multiple capture setting, and in the single capture setting, it outperforms other models in all metrics except PSNR. However, MarioNETte performs best in M-PSNR, which means it performs better in the face region compared to the baseline. The low CSIM obtained with NeuralHead-FF is indirect evidence against the capacity deficiency of AdaIN-based methods.
[0083] Other identity reenactments FIG. 35 shows the evaluation results of replaying other identities on CelebV, and FIG. 33 shows images generated from the proposed method and criteria. MarioNETte and MarioNETte+LT properly preserve the target identity and outperform other models in CSIM. The proposed method mitigates the identity preservation problem regardless of whether the driver is the same identity or not. NeuralHead-FF performs slightly better in terms of PRMSE and AUCON than MarioNETte, but the low CSIM of NeuralHead-FF means that it fails to preserve the target identity. The landmark transformer greatly improves the preservation of identity, although PRMSE and AUCON are slightly reduced. The reduction may be due to the fact that the PCA criteria for representation decomposition is not diverse enough to encompass the entire space of representations. Furthermore, the decomposition of identity and the representation itself is a significant issue, especially in a single capture setting. User Research Two types of user studies are conducted to evaluate the performance of the proposed model. Comparative Analysis Considering three example images of the target and an image of the driver, two images generated by different models were shown to human evaluators to select the higher quality image. Users were asked to evaluate the quality of the images in terms of (1) identity preservation, (2) reproduction of the driver's pose and facial expression, and (3) photorealism. We report the win rate of the baseline model compared to the proposed model. We believe that user-reported scores better reflect the quality of other models than other indirect measurements. Realism Analysis Similar to the user study protocol of Zakharov et al. (2019), three photos of the same person were presented to human raters. Of the three photos, two were taken from a video and the remaining one was generated by the model. Users were instructed to select the image that differed from the other two images in identity aspects within a time limit of 3 seconds. We report the trick percentages, which represent the identity preservation and photorealism of each model. In both studies, we sampled 150 examples from CelebV, uniformly distributed across 100 different raters.
[0084] FIG. 36 shows that this model is preferred over the conventional method and has a realism score with a large difference compared to the conventional method. As a result, it represents the ability of MarioNETte to generate realistic reenactments while preserving the target identity in terms of human cognition. A slight preference for MarioNETte over MarioNETte+LT is shown. This is because MarioNETte+LT has a higher identity preservation ability, although its expression transfer is slightly reduced, as shown in FIG. 35. The identity preservation ability of MarioNETte+LT exceeds all other models in realism score, i.e., it is almost twice as high as MarioNETte's score in the multiple capture setting, so the small reduction in expression transfer is not a significant issue. Ablation experiments Ablation tests were performed to investigate the effect of the proposed components. We compare the following configurations that replicate other identities while keeping everything else the same: (1) MarioNETte is the proposed method in which both image attention block and target feature alignment are applied. (2) AdaIN corresponds to a model similar to MarioNETte, where the image attention block is replaced by the AdaIN remaining block and the target feature alignment is omitted. (3) +Attention is MarioNETte with only the image attention block applied. (4) +Alignment is where only the target feature alignment is used. Figure 37 shows the results of the ablation test. Due to identity preservation (e.g., CSIM), AdaIN has difficulty combining style features that rely only on AdaIN remaining blocks. +Attention handles the appropriate coordinates and greatly mitigates the problem in both single-shot and multi-shot settings. +Alignment shows higher CSIM compared to +Attention, but has difficulty generating plausible images for unseen poses and expressions, resulting in worse PRMSE and AUCON. MarioNETte leverages both attention and target feature alignment and performs better than +Alignment on all metrics under consideration.
[0085] +Alignment, which relies entirely on target feature alignment for replay, easily fails due to large pose differences between the target and the driver. MarioNETte is able to overcome this. Given a single driver image along with three target images (Fig. 38A), +Alignment shows defects in the forehead (indicated by the arrow in Fig. 38B). This is because (1) it warps low-level features with a large pose input and (2) it merges features of multiple targets with various poses. MarioNETte, on the other hand, handles the situation properly by processing not only the appropriate spatial coordinates in the target image, but also the appropriate image among several target images. The attention map highlighting the area the image attention block focuses on is shown in white in Fig. 38A. MarioNETte processes the forehead and the appropriate target images (targets 2 and 3 in Fig. 38A) that have a similar pose as the driver. Related Technology Classical approaches to face reenactment include the use of explicit 3D modeling of human faces (Blanz and Vetter 1999), where typically the parameters of the driver and target 3D MMMs are calculated from single images and finally blended (Thies et al. 2015; Thies et al. 2016). Image warping is another popular approach, which rectifies the target image using estimated flows obtained from 3D models (Cao et al. 2013) or scarce landmarks (Averbuch-Elor et al. 2017). Face reenactment research has embraced the recent success of neural networks that explore various image-to-image translation structures (Isola et al. 2017), such as the work of Xu et al. (2017) and Wu et al. (2018) that combines cycle consistency loss (Zhu et al. 2017). A blend of the two approaches has also been studied. Kim et al. (2018) trained an image translation network that maps reenacted renderings of a 3D face model to a realistic output.
[0086] Recently, structures capable of fusing the style information of the target with the spatial information of the driver have been proposed. AdaIN (Huang and Belongie 2017; Huang et al. 2018; Liu et al. 2019) layers, attention mechanisms (Zhu et al. 2019; Lathuili`ere et al. 2019; Park and Lee 2019), deformation tasks (Siarohin et al. 2018; Dong et al. 2018), and GAN-based methods (Bao et al. 2018) have all been widely adopted. Similar ideas have been applied to the image-based face replay setting several times, including the use of AdaIN layers combined with image-level (Wiles, Koepke, and Zisserman 2018) and feature-level (Siarohin et al. 2019) warping, and meta-learning (Zakharov et al. 2019). The problem of identity mismatch has been studied by methods such as CycleGAN-based landmark transformer (Wu et al. 2018) and landmark swapper (Zhang et al. 2019). Although effective, these methods require independent models for each person or datasets containing image pairs, which are difficult to obtain. conclusion Here, we propose a framework for face replay in several iterations. The proposed image attention block and target feature alignment, together with the landmark transformation unit, can handle identity mismatches that occur using other people's landmarks. The proposed method does not require an additional fine-tuning step for identity adaptation, which significantly increases the usefulness of the model in real-world deployment. The experiments, including human evaluation, suggest the superiority of the proposed method. Future research directions include improving the landmark transformation part to better handle landmark decomposition, making the replay more convincing. Supplementary Material Detailed information on the MarioNETte structure
[0087] structural design Driver image x and K target image Given JPEG0007579674000085.jpg1033, the proposed several-fold face replay framework, called MarioNETte, first extracts a 2D landmark image (i.e., JPEG0007579674000086.jpg610 and Generates a 3D landmark detector. Using JPEG0007579674000088.jpg864 (Bulat and Tzimiropoulos 2017), JPEG0007579674000089.jpg936 and The facial key points are extracted, including information on pose and expression, as shown in JPEG0007579674000090.jpg835. After that, the 3D landmarks are rasterized into an image using the rasterizer R. I get JPEG0007579674000091.jpg967. We use a simple rasterizer that projects the 3D landmark points (e.g., (x, y, z)) orthogonally onto a 2D xY plane (e.g., (x, y)) and groups the projected landmarks into eight categories: left eye, right eye, contour, nose, left eyebrow, right eyebrow, inner mouth, and outer mouth. We draw lines between the points in a predefined order using predefined colors for each group (e.g., red, red, green, blue, yellow, yellow, cyan, and cyan, respectively). This results in the rasterized image shown in FIG. 39. MarioNETte is a conditional image generator JPEG0007579674000092.jpg987 and projection classifier The classifier D is constructed by the given image JPEG0007579674000094.jpg77 is the rasterized landmark Determine whether JPEG0007579674000095.jpg87 is an actual image of the distribution of data given the conditional input of identity c.
[0088] generator JPEG0007579674000096.jpg987 is further broken down into four components: the target encoder, the driver encoder, the blender, and the decoder. JPEG0007579674000097.jpg928 is the target feature map obtained by warping the target image. The target feature map z encoded along with JPEG0007579674000098.jpg95 y Generate the driver encoder JPEG0007579674000099.jpg821 receives the driver image and creates the driver feature map z x Generate Blender JPEG0007579674000100.jpg952 combines the encoded feature maps and creates the mixed feature map z xy Generates a decoder. JPEG0007579674000101.jpg1055 generates the reconstructed image. y are concatenated per channel and fed to the target encoder. Target Encoder JPEG0007579674000102.jpg928 employs a U-Net (Ronneberger, Fischer and Brox 2015) style structure that includes five downsampling blocks and four upsampling blocks using skip connections. The five feature maps generated by the downsampling blocks are The most downsampled feature map, s5, in JPEG0007579674000103.jpg931 is the encoded target feature map z y Used as the rest JPEG0007579674000104.jpg930 is converted to a normalized feature map. Normalized flow map JPEG0007579674000105.jpg959 has the following warping features: Using JPEG0007579674000106.jpg88, we normalize each feature map to a normalized feature map. Convert to JPEG0007579674000107.jpg1145.
number
[0089] We employ a bilinear sampler-based warping function, which is widely used with neural networks due to its differentiation potential (Jaderberg et al. 2015; Balakrishnan et al. 2018; Siarohin et al. 2019). Each s j Since the width and height are different, f y Let the size of S j To match the size of y applies to. Driver Encoder JPEG0007579674000109.jpg921 is composed of the four remaining downsampling blocks and contains the image of the driver's landmarks. x Take the driver's feature map z x Generate. Brenda JPEG0007579674000110.jpg853 is z x Location information and target style feature map z y Mixing and to create a mixed feature map z xyを Generate. Stack attention blocks on three images to create a blender. decoder JPEG0007579674000111.jpg957 is composed of four warp alignment blocks and the remaining upsampling blocks, the last of which is followed by an extra convolutional layer and a hyperbolic tangent activation function. Classifier JPEG0007579674000112.jpg829 consists of five remaining downsampling blocks without a self-attention layer. We employ a projection classifier with a slight modification to remove the global summation layer in the original structure. By removing the global summation layer, the classifier generates scores for multiple patches similar to the PatchGAN classifier (Isola et al. 2017). We construct the network by adopting the remaining upsampling and downsampling blocks proposed by Brock, Donahue, and Simonyan (2019). All collective normalization layers are replaced with instance normalization, except for the target encoder and the classifier, which do not have normalization layers. We utilize ReLU as the activation function. The number of channels whose output is downsampled (or upsampled) is doubled (or halved). The minimum number of channels is set to 64, and the maximum number of channels is set to 512 for all layers. The input images used as inputs for the target encoder, driver encoder, and classifier are first projected through a convolutional layer to match the channel size of 64.
[0090] Position Encoding We use a sinusoidal position encoding introduced by Vaswani et al. (2017) with some modifications. First, we split the number of channels for the position encoding in half. From now on, we use half of these to encode the horizontal coordinates and the rest to encode the vertical coordinates. To encode the relative positions, we normalize the absolute coordinates by the width and height of the feature map. Thus, Given the feature map for JPEG0007579674000113.jpg844, the corresponding position encoding JPEG0007579674000114.jpg847 is calculated as follows:
number
[0091] loss function Our model is adversarially trained using a projection classifier D (Miyato and Koyama 2018). The classifier aims to distinguish between real images of identity c and synthetic images of c generated by G. Since paired target and driver images of different identities cannot be obtained without explicit annotation, we train the model using target and driver images extracted from the same video sequence. Thus, x and y i The identity of is always the same for all target and driver image pairs during training (e.g., c). That is, JPEG0007579674000116.jpg838). We use the hinge GAN loss (Lim and Ye 2017) to optimize the discriminator D as follows:
number
number
number
[0092] Learning details To stabilize the adversarial learning, we apply spectral normalization (Miyato et al. 2018) to all layers of the classifier and generator. We also use the convex hull of the facial landmarks as a face region mask, and calculate the perceptual loss while giving a weighting value of 3 times to the position of the mask. We train the model using the Adam Optimizer, where we use a 2 × 10 -4 A learning rate of 5×10 is used for the classifier. -5 is used for the generator and style encoder. Unlike the setting in Brock, Donahue and Simonyan (2019), we update the discriminator only once per generator update. During training,、 λ P 10, λ PF 0.01, λ FM Set,to 10, and the number of target images,K,to 4. Detailed information on landmark conversion section Landmark Separation Formally, the landmark separation is computed as follows:
number
[0093] Landmark Decomposition Using the representation geometry obtained from the VoxCeleb1 training data, we evaluate the representation criterion b exp To compute, we divide the landmarks into other groups (e.g., left eye, right eye, eyebrows, mouth, etc.) and perform PCA on each group. We use PCA dimensions of 8, 8, 8, 16, and 8 for each group, resulting in a total of 48 representation criteria n exp get. The landmark decomposition part is trained separately on the VoxCeleb1 training set. Before training the landmark decomposition part, the parameters of each expression are normalized and the standard normal distribution is used to facilitate regression training. We followed the example of JPEG0007579674000134.jpg825. We use ResNet50 pre-trained on Imagenet (He et al. 2016) to extract features from the first layer to the last layer just before the global average pooling layer. The extracted image features are the average landmarks. JPEG0007579674000135.jpg88 subtracted normalized landmarks The network is concatenated with JPEG0007579674000136.jpg85 and fed into a two-layer MLP, after which ReLU activation is performed. The whole network is trained with a learning rate of 3 × 10 -4 The Adam optimization tool is used to optimize the predicted and target representation parameters by minimizing the MSE loss between them. During training, gradient clipping was used with a maximum gradient normal of 1. The representation strength parameter λ exp is set to 1.5.
[0094] Additional ablation experiments Quantitative results 34 and 35, MarioNETte outperforms NeuralHead-FF in PRMSE and AUCON in the VoxCeleb1 self-repeat setting, but this is reversed when replicating other identities in CelebV. We explain the above phenomenon through an ablation study. Figure 40 shows the evaluation results of the ablation model under the self-replay setting for VoxCeleb1. Unlike the evaluation results for other identity replays on CelebV (Figure 37 in this paper), +Alignment and MarioNETte outperform AdaIN in PRMSE and AUCON. This phenomenon may be due to the characteristics of the training dataset and other inductive biases of other models. VoxCeleb1 consists of short video clips (usually 5-10 seconds long) that exhibit similar poses and expressions between the driver and the target. Unlike the AdaIN-based model, which does not recognize spatial information, the proposed image attention block and target feature alignment encode the spatial information of the target image. This is presumably why the proposed model can overfit to similar identity pairs with similar pose and expression settings.
[0095] Qualitative results Figures 43 and 44 show the results of the ablation model recreating other identities in CelebV under single and multiple capture settings, respectively. AdaIN is unable to generate images similar to the target identity, but +Attention successfully preserves the main characteristics of the target. The target feature alignment module adds details to the generated images. However, while MarioNETte produces more natural images in several imaging settings, +Alignment does not easily handle images of multiple targets with various poses and expressions. Inference Time In this section, we report the inference time of our model. We measured the latency of the presented method while generating 256 × 256 images with different numbers of target images K ∈ {1, 8}. We ran each configuration 300 times and reported the average speed. We used an Nvidia Titan Xp and Pytorch 1.0.1.post2. As mentioned in this paper, we leveraged the open source realization of Bulat and Tzimiropoulos (2017) to extract 3D facial landmarks. Figure 41 shows the inference time analysis of the model. The total inference time of the proposed models MarioNETte+LT and MarioNETte can be derived as shown in Figure 3. The z y and JPEG0007579674000137.jpg98 is generated only once at the beginning. Therefore, it is inferred that it is divided into a target encoding part and a driver generated part. Since we perform batch inference on multiple target images, the inference time of the proposed components (e.g., target encoder and target landmark transformer) scales nonlinearly with the number of target images K. On the other hand, the open source 3D landmark detector processes images sequentially, so the processing time scales linearly.
[0096] Additional examples of generated images We provide additional qualitative results of the baseline methods and the proposed model on the VoxCeleb1 and CelebV datasets. Except for Monkey-Net, which is designed to use only a single image, we report qualitative results for the single-shot and multi-shot (8 target images) settings. In the multi-shot replay case, due to limited space, we only display one target image. Figures 45 and 46 compare different methods for self-replay of VoxCeleb1 in single-shot and multi-shot settings, respectively. Examples of single-shot and multi-shot replays with mismatched driver and target identities in VoxCeleb1 are shown in Figures 13 and 48. Qualitative results for the CelebV dataset are shown in Figures 49, 50, and 51. Various self-replay settings for single and multiple shots are compared in Figures 15 and 50. The results of replaying different identities from CelebV depending on the multiple shot setting can be seen in Figure 51. Figure 52 shows examples of failures generated by MarioNETte+LTd during single-shot replays under different identity settings in VoxCeleb1. The main cause of failure appears to be due to the large pose difference between the driver and the target. The above-described embodiments may be implemented in the form of a recording medium including computer-executable instructions, such as a program module executed by a computer. The computer-readable medium may be any available medium that can be accessed by a computer, and may include both volatile and non-volatile media, and removable and non-removable media. Computer-readable media may also include computer storage media, which may include all volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data.
[0097] At least one of the components, parts, modules, or units (collectively, in this paragraph, "components") represented by blocks in the drawings, such as in FIG. 2, FIG. 9, FIG. 17, FIG. 18, FIG. 23-FIG. 27, and FIG. 29-FIG. 32, may be realized as various hardware, software, and / or firmware structures that perform the respective functions described above according to the exemplary embodiment. For example, at least one of these components may use direct circuit structures such as memories, processors, logic circuits, look-up tables, etc. that may perform the respective functions through control by one or more microprocessors or other control devices. At least one of these components may also be specifically realized as a part of a module, program, or code including one or more executable instructions for performing a particular logic function, and may be executed by one or more microprocessors or other control devices. At least one of these components may also be realized by or include a processor, such as a central processing unit (CPU), that executes the respective functions, microprocessors, etc. Two or more of these components may be combined as one single component that performs all the operations or functions of the two or more components combined. Also, at least a part of the functions of at least one of these components may be performed by other components of these components. Also, although a bus is not shown in the above block diagram, communication between the components may be performed via a bus. Functional aspects of the above exemplary embodiments may be realized by algorithms executed on one or more processors. Also, the components represented by blocks or processing steps may use any number of related technologies, such as electronic configuration, signal processing and / or control, data processing, etc. Although the embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention can be embodied in other specific forms without changing the technical spirit or essential features of the present invention. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and are not limiting.
Claims
1. Extracting landmarks from each of a driver image and a target image; generating a driver feature map based on pose information and expression information of a first face appearing in the driver image; generating a target feature map and a pose-normalized target feature map based on style information of a second face appearing in a target image; generating a mixed feature map using the driver feature map and the target feature map; and generating a reenacted image using the mix feature map and the pose normalized target feature map.
2. 2. The method of claim 1, wherein the step of generating the driver feature map comprises inputting pose information and facial expression information corresponding to the first face into an artificial neural network to generate the driver feature map.
3. The method of claim 1 , wherein the landmarks include information regarding the location of at least one of the eyes, nose, mouth, eyebrows, and ears of the first face.
4. The method of claim 1 , wherein the target feature map includes style and pose information of the second face.
5. The method of claim 1 , wherein the pose-normalized target feature map corresponds to an output for the second facial style information input to an artificial neural network.
6. 2. The method of claim 1, wherein the step of generating the mix feature map comprises generating the mix feature map based on attention between the first facial pose information and expression information of the target feature map and the second facial style information of the driver feature map.
7. 2. The method of claim 1, wherein generating the mix feature map comprises encoding horizontal coordinates using half of the channels of positional encoding of the driver feature map and the target feature map and encoding vertical coordinates using the other half of the channels of positional encoding.
8. The method of claim 1 , wherein the style information includes at least one of texture information, color information, and shape information corresponding to the second face.
9. The method of claim 1 , wherein the mix feature map is generated such that the second facial landmarks have corresponding poses and expressions to the first facial landmarks.
10. The method of claim 1 , wherein the mix feature map is generated to reflect spatial information of the second face included in the target feature map.
11. The method of claim 1 , wherein the replayed image shows the identity of the second face and the pose and expression of the first face.
12. A computer-readable recording medium having a program recorded thereon for executing the method according to claim 1.
13. a landmark conversion unit that extracts landmarks from each of a driver image and a target image; a first encoder for generating a driver feature map based on pose information and expression information of a first face appearing in the driver image; a second encoder for generating a target feature map and a pose-normalized target feature map based on style information of a second face appearing in a target image; an image attention unit that uses the driver feature map and the target feature map to generate a mixed feature map; a decoder for generating a reenacted image using the mix feature map and the pose normalized target feature map.
14. The apparatus of claim 13 , wherein the first encoder inputs pose information and facial expression information corresponding to the first face into an artificial neural network to generate the driver feature map.
15. The apparatus of claim 13 , wherein the landmarks include information regarding the location of at least one of the eyes, nose, mouth, eyebrows, and ears of the first face.
16. The apparatus of claim 13 , wherein the target feature map includes style and pose information of the second face.
17. The apparatus of claim 13 , wherein the pose-normalized target feature map corresponds to an output for the second facial style information input to an artificial neural network.
18. 14. The apparatus of claim 13, wherein the image attention unit generates the mix feature map based on attention between the first facial pose and expression information of the target feature map and the second facial style information of a driver feature map.
19. 14. The apparatus of claim 13, wherein the image attention unit is configured to encode a horizontal coordinate using half of the channels of positional encoding of the driver feature map and the target feature map and to encode a vertical coordinate using the other half of the channels of positional encoding.
20. The apparatus of claim 13 , wherein the style information comprises at least one of texture information, color information, and shape information corresponding to the second face.
21. The apparatus of claim 13 , wherein the mix feature map is generated such that the second facial landmarks have a pose and expression that correspond to the first facial landmarks.
22. The apparatus of claim 13 , wherein the mix feature map is generated to reflect spatial information of the second face included in the target feature map.
23. The apparatus of claim 13 , wherein the replayed image shows the identity of the second face and the pose and expression of the first face.
24. At least one processor; and a memory for storing at least one instruction for execution by said at least one processor; The at least one processor Extracting landmarks from each of the driver image and the target image; generating a driver feature map based on pose information and expression information of a first face appearing in the driver image; generating a target feature map and a pose-normalized target feature map based on style information of a second face appearing in a target image; generating a mixed feature map using the driver feature map and the target feature map; A mobile device generating a reenacted image using the mix feature map and the pose normalized target feature map.
Citation Information
Patent Citations
graphics and image processing system
JP2002525764A
Real-time Expression Transfer for Facial Reenactment
US20180068178A1
Neural face editing with intrinsic image disentangling
US20180365874A1
