Method and computing device for face reproduction

Abstract

A method and computing device for realistic real-time face rendering are provided. An example method includes receiving, by a computing device, visual data including a visible portion of a source face; determining, by the computing device and based on the visible portion of the source face, a first portion of a source face parameter associated with the parameterized face model, the first portion corresponding to the visible portion; predicting, by the computing device and in part based on the visible portion of the source face, a second portion of the source face parameter, the second portion corresponding to a remaining portion of the source face; receiving, by the computing device, a target video, the target video comprising a target face; determining, by the computing device, a target face parameter associated with the parameterized face model and corresponding to the target face based on the target video; and synthesizing, by the computing device, an output face based on the source face parameters and the target face parameters using the parameterized face model, the output face including the source face that mimics the facial expression of the target face.

Description

[0001] This application is a divisional application of the patent application with application number 202080007533.6 and invention name “System and method for facial reproduction”, which entered the national stage on June 28, 2021, and has an international application date of January 18, 2020 and international application number PCT / US2020 / 014219. The entire contents of the application are incorporated herein by reference. Technical Field

[0002] The present disclosure relates generally to digital image processing and more particularly to methods and systems for facial reproduction. Background Art

[0003] Facial reproduction can include transferring the facial expressions of a source individual in a source video to a target individual in a target video or target image. Facial reproduction can be used for manipulation and animation of faces in many applications, such as entertainment programs, computer games, video conversations, virtual reality, augmented reality, etc.

[0004] Some current techniques for facial reproduction use deformable facial models to re-render a target face with different facial expressions. Although faces with deformable facial models can be generated quickly, the generated faces may not be realistic. Some other current techniques for facial reproduction can re-render a target face based on the use of deep learning methods. Deep learning methods can allow realistic results to be obtained. However, deep learning methods are time-consuming and may not be suitable for real-time facial reproduction on conventional mobile devices. Summary of the invention

[0005] This section briefly introduces a selection of concepts that are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] According to one embodiment of the present disclosure, a method for facial reproduction is provided. The method may include receiving a target video through a computing device. The target video may include at least one target frame. At least one target frame may include a target face. The method may also include receiving a source video through a computing device. The source video may include a source face. The method may also include determining at least one target facial expression through a computing device and based on a target face in at least one target frame of the target video. The method may also include determining at least one source facial expression through a computing device and based on a source face in a frame of the source video. The method may also include synthesizing an output face through a computing device and using a parameterized facial model. The output face may include a target face in which the target facial expression is modified to mimic the source facial expression. The method may also include generating a mouth area and an eye area through a computing device and based on a deep neural network (DNN). The method may also include combining the output face, the mouth area, and the eye area through a computing device to generate a frame of an output video.

[0007] In some embodiments, the parameterized facial model may depend on facial expression, facial identity, and facial texture.

[0008] In some embodiments, the parameterized facial model may include a template mesh pre-generated based on historical images of faces of multiple individuals. The template mesh may include a predetermined number of vertices. In certain embodiments, the texture model includes a set of colors associated with the vertices.

[0009] In some embodiments, the individuals can be of different ages, genders, and races. In certain embodiments, the historical images of faces can include at least one set of photos of a single individual having a predetermined number of facial expressions. The facial expressions include at least one of a neutral expression, an open mouth expression, a smiling expression, and an angry expression.

[0010] In some embodiments, the parameterized facial model may also include a set of blend shapes, the blend shapes representing facial expressions.

[0011] In some embodiments, the input to the DNN may include at least parameters associated with a parameterized facial model.

[0012] In certain embodiments, the input of the DNN may include a previous mouth region and a previous eye region. The previous mouth region and the previous eye region can be associated with at least one previous frame of the target video.

[0013] In some embodiments, the DNN can be trained using historical images of faces of multiple individuals.

[0014] According to another embodiment, a system for facial reproduction is provided. The system may include at least one processor and a memory storing processor executable code, wherein the at least one processor can be configured to implement the operations of the above-mentioned method for facial reproduction when executing the processor executable code.

[0015] According to yet another embodiment of the present disclosure, a non-volatile processor-readable medium storing processor-readable instructions is provided. When the processor-readable instructions are executed by a processor, the processor implements the above-mentioned method for facial reproduction.

[0016] According to an exemplary embodiment, a system for providing personalized advertisements is disclosed. The system may include a database configured to store more than one advertisement video. The advertisement video may include at least a target face. The target face can be associated with a first individual. The system may also include a user information collection module configured to receive user data associated with a user. The user data may include at least an image of a source face. The source face can be associated with a second individual different from the first individual. The user information collection module can determine parameters of the source face based on the user data. The system may also include a personalized video generation module. The personalized video generation module can be configured to segment more than one advertisement video into a first part and a second part. The first part can include the target face, and the second part can include a background. The personalized video generation module can modify a first part of a frame based on the parameters of the source face to replace the target face with the source face. The personalized video generation module can also combine the modified first part with the second part to obtain an output frame of the output advertisement video.

[0017] The frames of the advertisement video stored in the database can be pre-processed to be segmented into a first portion including a target face and a second portion including a background.

[0018] The parameters of the source face may include a source facial expression, a source facial identity, and a source facial texture. Modifying the first part of the frame may include: determining a target facial expression, a target facial identity, and a target facial texture based on the target face, and replacing the target facial identity with the source facial identity and replacing the target facial texture with the source facial texture. The frame of the advertisement stored in the database can be preprocessed to include the parameters of the target face, the parameters of the target face including the target facial expression, the target facial identity, and the target facial texture.

[0019] The user information collection module can be configured to read user data from a computing device of the user or one or more accounts of the user on one or more social media platforms. The user data may include information about the user's interests and one or more images of the user, one or more friends of the user, and one or more favorite stars of the user. The user information collection module can analyze the user data to select one or more images of the user, one or more friends of the user, or one or more favorite stars of the user. The user information collection module can determine parameters of the source face based on the selected one or more images.

[0020] The user information collection module can be configured to receive user input from a user. The user input can represent a list of images to be used for replacement of the target face. The user information collection module can determine parameters of the source face based on the list of images.

[0021] The personalized video generation module can be configured to publish the output advertisement video to one or more accounts of the user in one or more social media platforms.

[0022] According to another exemplary embodiment, a method for providing personalized advertising is disclosed. The method may include storing one or more advertising videos in a database. The advertising video may include a target face associated with a first individual. The method may include receiving user data associated with a user through a user information collection module. The user data may include an image of at least one source face. The source face can be associated with a second individual different from the first individual. The method may determine parameters of the source face through the user information collection module and based on the user data. The method may also include segmenting a frame of one or more advertising videos into a first part and a second part through a personalized video generation module. The first part may include the target face and the second part may include a background. The method may include modifying a first part of the frame through a personalized video generation module and based on the parameters of the source face to replace the target face with the source face. The method may also include combining the modified first part and the second part through the personalized video generation module to obtain an output frame of the output advertising video.

[0023] Other purposes, advantages and novel features of the examples will be set forth in part in the following description, and in part will become apparent to those skilled in the art upon examination of the following description and the accompanying drawings, or may be learned by production or operation of the examples. The purposes and advantages of the concepts may be realized and obtained by the methods, means and combinations particularly pointed out in the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like reference numerals indicate similar elements.

[0025] Figure 1 is a block diagram illustrating an exemplary environment in which a method for facial reproduction can be implemented.

[0026] Figure 2 is a block diagram illustrating an exemplary embodiment of a computing device implementing a method for facial reproduction.

[0027] Figure 3 is a schematic diagram illustrating a process of face reproduction according to an exemplary embodiment.

[0028] Figure 4 is a block diagram illustrating a system for face reproduction according to an exemplary embodiment.

[0029] Figure 5 A process flow diagram of a method for facial reproduction according to an exemplary embodiment is shown.

[0030] Figure 6 is a schematic diagram illustrating generation of a personalized video according to an exemplary embodiment.

[0031] Figure 7 is a block diagram of a system for providing personalized advertisements according to some exemplary embodiments of the present disclosure.

[0032] Fig. 9 An exemplary computer system that can be used to implement the method of facial reproduction is shown. DETAILED DESCRIPTION

[0033] The following detailed description of the embodiments includes reference to the accompanying drawings, which form a part of the detailed description. The methods described in this section are not prior art to the claims and are not considered prior art because they are included in this section. The accompanying drawings show diagrams according to exemplary embodiments. These exemplary embodiments (also referred to as "examples" herein) are described in sufficient detail to enable those skilled in the art to practice this subject matter. Without departing from the scope of the claims, the embodiments can be combined, other embodiments can be utilized, or structural, logical and operational changes can be made. Therefore, the following detailed description is not restrictive, and the scope is defined by the attached claims and their equivalents.

[0034] The present disclosure relates to methods and systems for facial reproduction. The embodiments provided in the present disclosure solve at least some of the problems in the prior art. The present disclosure can be designed to work in real time on a mobile device such as a smartphone, tablet or mobile phone, and does not require connection to the Internet or use of server-side computing resources, although the embodiments can be extended to methods involving network services or cloud-based resources.

[0035] Some embodiments of the present disclosure may allow a target face in a target video or target image to be animated using facial expressions belonging to a source face in a source video. The target face can be manipulated in real time by the facial expressions of the source face. Embodiments of the present disclosure include: using previously recorded facial images of different individuals to establish a statistical facial deformable model, training a deep learning model to synthesize mouth and eye areas, and performing real-time facial reproduction. Some embodiments may significantly reduce the computation time for realistic facial reproduction and allow realistic facial reproduction on mobile devices.

[0036] Some embodiments of the present disclosure may allow for generating personalized ads for users by replacing the faces of actors in ads with faces of other individuals. Even when specifically targeted to users, standard ads are often skipped by users because they may not attract the user's attention or may not be easy to remember. However, if an ad features one of the user's friends or favorite stars, such an ad may attract the user's attention and cause the user to watch the ad until the end. Various techniques can be used to implement the present disclosure. For example, the methods described herein can be implemented by software running on a computer system, and / or by hardware utilizing a combination of microprocessors or other specially designed application-specific integrated circuits (ASICs), programmable logic devices, or combinations thereof. In particular, the methods described herein can be implemented by a series of computer executable instructions residing on a non-volatile storage medium such as a disk drive or a computer-readable medium. It should be noted that the methods disclosed herein can be implemented by computing devices such as mobile devices, personal computers, servers, network nodes, etc.

[0037] For the purposes of this patent application, the terms "or" and "and" shall mean "and / or" unless otherwise specified or the context in which they are used clearly dictates otherwise. The term "a" shall mean "one or more" unless otherwise specified or the use of "one or more" is clearly inappropriate. The terms "include", "comprising" and "including" are interchangeable and are not limiting. For example, the term "include" shall be interpreted as "including but not limited to".

[0038] According to one embodiment of the present disclosure, an example method for facial reproduction may include receiving a target video by a computing device. The target video may include at least one target frame. At least one target frame may include a target face. The method may also include receiving a source video by a computing device. The source video may include a source face. The method may also include determining at least one target facial expression by a computing device and based on a target face in at least one target frame of the target video. The method may also include determining at least one source facial expression by a computing device and based on a source face in a frame of the source video. The method may also include synthesizing an output face by a computing device and using a parameterized facial model and a texture model. The output face may include a target face in which the target facial expression is modified to mimic the source facial expression. The method may also include generating a mouth area and an eye area by a computing device and based on a DNN. The method may also include combining the output face, the mouth area, and the eye area by a computing device to generate a frame of the output video.

[0039] Exemplary embodiments are now described with reference to the accompanying drawings. The accompanying drawings are schematic diagrams of ideal exemplary embodiments. Thus, it should be apparent to those skilled in the art that the exemplary embodiments discussed herein should not be construed as being limited to the specific illustrations presented herein, and that these exemplary embodiments can include differences and be different from the illustrations presented herein.

[0040] Figure 1An exemplary environment 100 is shown in which a method for facial reproduction can be practiced. The environment 100 may include a computing device 110, a user 130, and a cloud-based computing source 170 (also referred to as a computing cloud 170). The computing device 110 may include a camera 115 and a graphics display system 120. The computing device 110 may refer to a mobile device such as a mobile phone, a smart phone, or a tablet computer. However, in other embodiments, the computing device 110 may refer to a personal computer, a laptop computer, a netbook, a set-top box, a television device, a multimedia device, a personal digital assistant, a game console, an entertainment system, an infotainment system, a vehicle computer, or any other computing device. The computing device 110 can be connected to the computing cloud 170 via a digital network communication. The cloud-based computing source 170 can include computing sources (hardware and software) that are available at a remote location and can be accessed through a network (e.g., the Internet). The cloud-based computing source can be shared by multiple users and can be dynamically reallocated based on demand. The cloud-based computing source can include more than one server farm / cluster, which includes a collection of computer servers that can be co-located using network switches and / or routers. In some embodiments of the present disclosure, computing device 110 can be configured to display target video 125. Target video 125 can include at least one frame that includes target face 140. In some embodiments, target video 125 can include a single image. In some embodiments, target video 125 can be pre-recorded and stored in a memory of computing device 125 or in a cloud-based computing source to which computing device 125 is communicatively connected.

[0041] In certain embodiments, computing device 110 may be configured to capture source video via, for example, camera 115. The source video may include at least the face of user 110 (also referred to as the source face). In some other embodiments, the source video can be stored in a memory of computing device 110 or in computing cloud 170.

[0042] In some other embodiments, some target videos or images can be pre-recorded and stored in the memory of the computing device 110 or in the computing cloud 170. The user can select the target video or image to be manipulated, and a source video to be used to manipulate the target video or image. According to various embodiments of the present disclosure, the computing device 110 can be configured to analyze the source video to extract the parameters of the facial expression of the user 110. The computing device 110 can be further configured to modify the target video 125 based on the parameters of the facial expression of the source face, so that the target face 140 repeats the facial expression of the source face in real time. In other embodiments, the computing device can also be configured to modify the target video 125 so that the target face 140 repeats the voice of the user 110.

[0043] In some embodiments of the present disclosure, the computing device may be configured to receive user input. The user input may include one or more scenarios indicating how to control the facial parameters of the target face. The scenario may include a series of facial expressions and types of movements of the target face 140 that the user 130 wishes to see in the modified target video 125. The user input may also include an environment variable indicating the type of computing device used to generate the modified video.

[0044] In some other embodiments of the present disclosure, computing device 110 or cloud-based computing source 170 may store one or more images of user 110. The image may include the face of user 110. The image may also include a set of photos or a set of videos taken under different conditions. For example, photos and videos may be taken from different angles and under different lighting conditions relative to the face of user 130. In some embodiments, computing device 110 or computing cloud 170 may store one or more images of other individuals (e.g., friends of user 110 or favorite celebrities of user 110).

[0045] According to some embodiments of the present disclosure, computing device 110 or cloud-based computing source 170 can be configured to analyze the stored image of user 130 to extract facial parameters of user 130. Computing device 110 or cloud-based computing source 170 can also be configured to modify target video 125 by replacing target face 140 in target video 125 with the face of user 130 based on the facial parameters of user 130.

[0046] Similarly, computing device 110 or cloud-based computing source 170 can be configured to analyze stored images of user 130 to extract facial parameters of other individuals (e.g., friends of user 130 or user 130's favorite celebrity). Computing device 110 can also be configured to modify target video 125 by replacing target face 140 in target video 125 with the individual's face based on the individual's facial parameters. In some embodiments, computing device 110 or cloud-based computing source 170 can be configured to keep the facial expression of target face 140 unchanged while replacing the target face with the face of user 130 or other individuals. Figure 2 In the example shown, the computing device 110 includes hardware components and software components. In particular, the computing device 110 includes a camera 115 or any other image capture device or scanner to acquire digital images. The computing device 110 can also include a processor module 210 and a storage module 215 for storing software components and processor-readable (machine-readable) instructions or codes, which, when executed by the processor module 210, cause the computing device 200 to perform at least some steps of the method for facial reproduction as described herein.

[0047] The computing device 110 can also include a facial rendering system 220, which can include hardware components (e.g., separate processing modules and memory), software components, or a combination thereof. The facial rendering system 220 can be configured to perform facial rendering as described herein. Figure 4 The facial reproduction system 220 is described in more detail.

[0048] The computing device 110 can also include a system 700 for providing personalized advertising. Figure 7 System 700 is described in more detail.

[0049] Figure 3 3 is a schematic diagram illustrating the functionality 300 of the facial reproduction system 220 according to an exemplary embodiment. According to various embodiments of the present disclosure, the facial reproduction system 220 may receive a source video 305 and a target video 125. The source video 305 may include more than one frame 310. The frame 310 may include a source face 315. The target video 125 may include more than one frame 320. The frame 320 may include a target face 140. In some embodiments, the target video 125 may include only one frame (in other words, a single photo). The facial expressions of the source face 315 and the target face 140 may be different.

[0050] In some embodiments of the present disclosure, the facial reproduction system 220 can be configured to analyze the frame 310 of the source video 305 to extract the source parameters 330. The source parameters 330 can be extracted by fitting a parameterized facial model to the source face 315. The parameterized facial model can include a template mesh. The coordinates of the vertices in the template mesh can depend on two parameters: facial identity and facial expression. Therefore, the source parameters 330 can include the facial identity and facial expression corresponding to the source face 315. The source parameters 330 can also include the texture of the source face 315. The texture can include the color of the vertices in the template mesh. In some embodiments, the texture model associated with the template mesh can be used to determine the texture of the source face 315.

[0051] In some embodiments of the present disclosure, the facial reproduction system 220 can be configured to analyze the frames 320 of the target video 305 to extract target parameters 335. The target parameters 335 can be extracted by fitting a parameterized facial model to the target face 140. The target parameters 335 can include a facial identity and a facial expression corresponding to the target face 140. The source parameters 330 can also include a texture of the target face 140. The texture of the target face 140 can be derived using a texture model.

[0052] In some embodiments of the present disclosure, the facial reproduction system 220 can also be configured to replace the facial expression in the target parameters 335 with the facial expression from the source parameters 330. The facial reproduction system 220 can also be configured to synthesize an output face 350 using the parameterized facial model, the texture model, and the target parameters 335 with the replaced facial expression. The output face 350 can be used to replace the target face 140 in a frame of the target video 125 to obtain a frame 345 of the output video 340.

[0053] Figure 4 4 is a block diagram of a facial reproduction system 220 according to an exemplary embodiment. The facial reproduction system 220 can include a parameterized facial model 405, a texture model 410, a DNN 415, a pre-processing module 420, a parameter extraction module 425, a facial synthesis module 425, and a mouth and eye generation module 430. The modules 405-430 can be implemented as software components for use with a hardware device such as a computing device 110, a server, or the like.

[0054] In some embodiments of the present disclosure, the parameterized facial model 405 can be pre-generated based on images of a predefined number of individuals of different ages, genders, and ethnic backgrounds. For each individual, the images may include an image of the individual with a neutral facial expression and one or more images of the individual with different facial expressions. Facial expressions may include open mouth, smiling, angry, surprised, etc.

[0055] The parameterized face model 405 may include a template mesh having a predefined number of vertices. The template mesh may be represented as a 3D triangulation defining the shape of the head. Each individual may be associated with an individual-specific blend shape. The individual-specific blend shapes may be adjusted to the template mesh. The individual-specific blend shapes may correspond to specific coordinates of vertices in the template mesh. Thus, different images of an individual may correspond to a template mesh of the same structure, however, the coordinates of the vertices in the template mesh may be different for different images.

[0056] In some embodiments of the present disclosure, the parameterized facial model may include a bilinear facial model that depends on two parameters, namely, facial identity and facial expression. The bilinear facial model can be established based on a blend shape corresponding to an image of an individual. Thus, the parameterized facial model includes a template mesh of a predetermined structure, wherein the coordinates of the vertices depend on the facial identity and the facial expression.

[0057] In some embodiments of the present disclosure, the texture model 410 can include a linear space of texture vectors corresponding to the image of the individual.The texture vectors can be determined as the colors at the vertices of the template mesh.

[0058] A face can be synthesized based on known parameters of facial identity, facial expression, and texture using parameterized facial model 405 and texture model 410. Unknown parameters of facial identity, facial expression, and texture can also be determined based on a new image of a new face using parameterized facial model 405 and texture model 410.

[0059] It is not time-consuming to synthesize a face using the parameterized face model 405 and the texture model 410; however, the synthesized face may not be realistic, especially the mouth and eye areas. In some embodiments of the present disclosure, the DNN 415 can be trained to generate realistic images of the mouth and eye areas of the face. The DNN 415 can be trained using a video collection of a speaking individual. The mouth and eye areas of the speaking individual can be captured from the frames of the video. The DNN 415 can be trained using a generative adversarial network (GAN) to predict the mouth and eye areas of the face based on a predetermined number of previous frames of the mouth and eye areas and the facial expression of the expected current frame. The parameters of the facial expression of the previous frames of the mouth and eye areas can be extracted at a specific moment. The DNN 415 can allow the mouth and eye areas to be synthesized using the parameters of the expected facial expression. The DNN 415 can also allow the spatial coherence to be obtained using previous frames.

[0060] The GAN adjusts the mouth and eye regions presented from the facial model, the current expression parameters, and embeds features from previously generated images, and makes the same regions more realistic. The mouth and eye regions generated using the DNN 415 can be used to replace the mouth and eye regions synthesized by the parameterized facial model 405. It should be noted that synthesizing the mouth and eye regions by the DNN can consume less time than synthesizing the entire face by the DNN. Therefore, for example, the mouth and eye regions can be generated using the DNN in real time by one or more processors of a mobile device such as a smartphone or tablet.

[0061] In some embodiments, the pre-processing module 420 can be configured to receive a target video 125 and a source video 305. The target video 125 can include a target face, and the source video can include a source face. The pre-processing module 420 can also be configured to segment at least one frame of the target video to obtain an image of the target face 140 and a target background. The segmentation can be performed using a neural network, a mask, and smoothing. Similarly, the pre-processing unit can be configured to segment a frame of the source video 305 to obtain an image of the source face 315 and a source background.

[0062] In some embodiments, the pre-processing module 420 can also be configured to determine a target parameter set based on at least one frame of the target video 125 using the parameterized facial model 405 and the texture model 410. In some embodiments, the target parameter set can include a target facial identity, a target facial expression, and a target texture. In some embodiments, the pre-processing module 420 can also be configured to determine a source parameter set based on at least one frame of the source video 305 using the parameterized facial model 405 and the texture model 410. The source parameter set can include a source facial identity, a source facial expression, and a source texture.

[0063] In some embodiments, the facial synthesis module 425 can be configured to transfer the facial expression of the source face 310 in the frame of the source video 305 to the target face 140 of the target video. In some embodiments, the facial synthesis module 425 can be configured to determine an output parameter set associated with the parameterized facial model. In a specific embodiment, a deformation transfer can be applied to the target parameter set to interpret the facial expression of the source face 310 and thereby obtain an output parameter set.

[0064] In some embodiments, the face synthesis module 425 can be configured to replace the target facial expression in the target parameter set with the source facial expression to obtain the output parameter set. The face synthesis module 425 can also be configured to synthesize the output face using the output parameter set and the parameterized face model 405 and the texture model 4410.

[0065] In some embodiments, a two-dimensional (2D) deformation can be applied to a target face to obtain a realistic image of a region of the output face hidden in the target face. Parameters of the 2D deformation can be determined based on a source parameter set of a parameterized face model.

[0066] In some embodiments, the mouth and eye generation module 430 can be configured to generate mouth and eye regions using the DNN 415 based on the source facial expression and at least one previous frame of the target video 125. The mouth and eye generation module can also be configured to replace the mouth and eye regions in the output face synthesized by the parameterized face model 405 and the texture model 410 with the mouth and eye regions synthesized by the DNN 415.

[0067] In other embodiments, the facial reproduction system 220 may include: a module for changing the background in the target video, a module for changing the hairstyle or appearance of the target face, and a module for adding visual effects in the target video. Some other embodiments may include transmitting other parameters from the source video 305 to the target video 125, and the other parameters include but are not limited to head movement, body posture and orientation, and speech.

[0068] Figure 5is a flow chart illustrating a method 500 for facial reproduction according to an exemplary embodiment. The method 500 can be performed by the computing device 110.

[0069] Method 500 can begin in block 505 by receiving a target video via computing device 110. The target video can include at least one target frame. The at least one target frame can include a target face.

[0070] In block 510, method 500 may include receiving, by computing device 110, a source video. The source video may include a source face.

[0071] In block 515 , method 500 may include determining, by computing device 110 and based on a target face in at least one frame of a target video, at least a target facial expression.

[0072] In block 520 , method 500 may include determining, by computing device 110 and based on a source face in a frame of a source video, at least a source facial expression.

[0073] In block 525, method 500 may include synthesizing, by computing device 110 and using parameterized model and texture model 410, an output face. The output face may include a target face in which the target facial expression is modified to mimic the source facial expression.

[0074] In block 530 , the method 500 may include generating, by the computing device and based on the DNN 415 , a mouth region and an eye region.

[0075] In block 535 , method 500 may include combining, by computing device 110 , the output face, mouth region, and eye region to generate a frame of an output video.

[0076] Figure 6 6 is a schematic diagram illustrating the generation of a personalized video according to an exemplary embodiment. Figure 6 In the example of , target video 610 includes an advertisement featuring actor 615. Facial reproduction system 220 can be used to replace the face of actor 615 in the advertisement with the face of individual 625 in source image 620. Output video 630 may include the advertisement shown in target video 610, wherein the face of actor 615 is replaced by the face of individual 625 in source image 620.

[0077] Figure 77 is a block diagram of a system 700 for providing personalized advertisements according to some exemplary embodiments of the present disclosure. The system 700 may include a video database 710, a user information collection module 720, and a personalized video generation module 730. The modules of the system 700 can be implemented as instructions stored in a memory of the computing device 110 and / or the computing cloud 170 and executed by one or more processors of the computing device 110 and / or the computing cloud 170. For example, the system 700 can be implemented in a mobile application, a desktop application, or a web-based application for one or more social networks.

[0078] The video database 710 can store more than one video. The video can include pre-stored high-quality video advertisements featuring one or more actors. The video can include 2D videos or 3D scenes. The video can be pre-processed to segment the actor's face and background in each frame and identify a parameter set that can be used to further insert a source face in place of the actor's face. The parameter set can include facial texture, facial expression, facial color, facial identity, facial position and angle, etc. The parameter set can also include a list of manipulations and operations that can be performed on the actor's face, such as replacement of the actor's face in a realistic manner.

[0079] The user information collection module 720 can receive information about the individual to be used for personalization. In some embodiments, the user information collection module 720 can receive visual information, such as one or more photos or one or more videos of the individual. Preferably, the photos and videos can be taken under different conditions, such as different positions and angles relative to the individual's face and different lighting conditions. The user information collection module 720 can process one or more photos or one or more videos of the individual to determine a parameter set associated with the individual. The parameter set may include facial texture, facial expression, facial color, and facial identity associated with the individual. In some embodiments, the module 720 can generate hidden parts of the face of the individual that are covered in the individual's photo or video. For example, the individual can wear glasses. It is possible to use DNN to generate hidden parts or parameters associated with hidden parts (facial texture, facial expression, facial color, and facial identity. It is possible to use GAN to train DNN to predict hidden parts of the individual's face, for example, mouth area, eye area, and head area.

[0080] In some embodiments, the system 700 can be used for social networks. In these embodiments, the information collection module 720 can obtain information about individuals from the user's social network account. The information obtained from the social network can include photos and videos of the user, a list of the user's friends and interests, and photos of the user's friends. The user can also upload photos and videos to be used to replace the actors in the advertisement. The user can also select photos and videos to be used to replace the faces of the actors in the advertisement. In a specific embodiment, the information collection module 720 can obtain information such as photos and videos of the user, a list of the user's friends and interests, and photos of the user's friends from data stored in the user's computing device 110. In some embodiments, the module 700 can (based on the list of the user's friends and interests) select individuals whose faces will be shown in the advertisement. The individual can be the user, one of the user's friends, or the user's favorite star.

[0081] The personalized video generation module 730 can generate personalized advertisements based on one or more videos stored in the database 710 based on user information. In some embodiments, the module 730 can automatically determine the user information for generating personalized advertisements and display the obtained personalized advertisements to the user. In a specific embodiment, the user can indicate the user information to be used (particularly a specific photo or video of an individual) and request the generation of personalized advertisements by inserting the face of the selected individual into the video. For example, a user can create a personalized advertisement using the user's own face and then post it on a social network so that all followers of the user can see the advertisement featuring the user. The module 730 can also store the personalized advertisement in the memory of the user's computing device 110 according to the user's request. The module 730 can also receive information about the viewing frequency of personalized advertisements.

[0082] Module 730 may replace the faces of actors in advertisements with the faces of selected persons by using a facial reconstruction system, as described above with reference to Figure 3 and Figure 4 As described. Module 730 can replace the facial texture, facial color, and facial identity of the actor with the facial texture, facial color, and facial identity of the selected person. Module 730 can also transfer the user's hair, clothing, etc. to the new image or video. In the new image or video, module 730 can also add an image of glasses to the eye area of ​​the selected person's face. Similarly, in the new image or video, module 730 can add an image of headwear (e.g., a hat, a helmet, etc.) to the head of the selected person. The images of glasses and headwear can be pre-stored in the user's computing device 110 or generated. The images of glasses and headwear can be generated using a DNN. Module 730 can also apply color to the face in the new image. For example, module 730 can add a tan to the face of the selected person.

[0083] Figure 8 is a flowchart illustrating a method 500 for providing personalized advertisements according to some exemplary embodiments of the present disclosure. The method 800 can be performed by Figure 7 The system 700 described in is executed.

[0084] Method 800 may begin in block 805 and store one or more advertisement videos in a database. The advertisement video may include at least a target face associated with a first individual.

[0085] In block 810, method 700 may continue to receive user data associated with the user via the user information collection module. The user data may include an image of at least one source face. The source face may be associated with a second individual different from the first individual.

[0086] In block 815 , method 800 may include determining, through the user information collection and based on the user data, parameters of the source face.

[0087] In block 820, the method 800 may segment the frame of the advertisement video into a first part and a second part by the personalized video generation module. The first part may include the target face, and the second part may include the background.

[0088] In block 825 , the method 800 may modify the first portion of the frame by personalizing the video generation module and based on the parameters of the source face to replace the target face with the source face.

[0089] In optional block 830, method 800 may apply one or more special effects to the modified first portion of the frame by the personalized video generation module. For example, method 800 may apply a tan to the face in the modified first portion, add glasses to the eye area of ​​the face or the head, and / or add a hat, helmet, or other headwear to the head or face.

[0090] In block 835 , the method 800 may combine the modified first portion with the second portion through the personalized video generation module to obtain an output frame of the output advertisement video.

[0091] Fig. 9 An example computing system 900 is shown that can be used to implement the methods described herein. The computing system 900 can be implemented in the context of the computing device 110, the facial reproduction system 220, the parameterized facial model 405, the texture model 410, the DNN 415, the pre-processing module 420, the facial synthesis module 425, and the mouth and eye generation module 430.

[0092] like Fig. 9As shown in , the hardware components of the computing system 900 may include one or more processors 910 and memory 920. The memory 920 stores, in part, instructions and data for execution by the processor 910. When the system 900 is running, the memory 920 can store executable code. The system 900 may also include: an optional mass storage device 930, an optional removable storage medium drive 940, one or more optional output devices 950, one or more optional input devices 960, an optional network interface 970, and one or more optional peripheral devices 980. The computing system 900 can also include one or more software components 995 (e.g., a software component capable of implementing the method for facial reproduction as described herein).

[0093] exist Fig. 9 The components shown in the figure are depicted as being connected via a single bus 990. The components may be connected via more than one data transmission means or data network. The processor 910 and the memory 920 may be connected via a local microprocessor bus, and the mass storage device 930, the peripheral device 980, the removable storage device 940, and the network interface 970 may be connected via more than one input / output (I / O) bus.

[0094] The mass storage device 930, which may be implemented by a magnetic disk drive, solid state disk drive, or optical disk drive, is a non-volatile storage device for storing data and instructions used by the processor 910. The mass storage device 930 can store system software (e.g., software component 995) for implementing the embodiments described herein.

[0095] Removable storage media drive 940 operates in conjunction with removable nonvolatile storage media, such as compact disks (CDs) or digital video disks (DVDs), to input data and code into and output data and code from computing system 900. System software (e.g., software components 995) used to implement the embodiments described herein may be stored on such removable media and input into computing system 900 via removable storage media drive 940.

[0096] Optional input device 960 provides a portion of the user interface. Input device 960 may include an alphanumeric keyboard such as a keyboard for inputting alphanumeric and other information, or a pointing device such as a mouse, trackball, stylus, or cursor direction keys. Input device 960 can also include a camera or scanner. In addition, in Fig. 9 The system 900 shown in FIG. 9 includes an optional output device 950. Suitable output devices include speakers, printers, network interfaces, and monitors.

[0097] The network interface 970 can communicate with external devices, external computing devices, servers, and network systems via one or more communication networks, such as one or more wired, wireless, or fiber optic networks, including, for example, the Internet, an intranet, a LAN, a WAN, a cellular phone network, a Bluetooth radio, and a radio frequency network based on IEEE 802.11, etc. The network interface 970 can be a network interface card (e.g., an Ethernet card), an optical transceiver, a radio frequency transceiver, or any other type of device capable of sending and receiving information. The optional peripheral devices 980 can include any type of computer support device to add additional functionality to the computer system.

[0098] The components included in the computing system 900 are intended to represent a broad class of computer components. Thus, the computing system 900 can be a server, a personal computer, a handheld computing device, a phone, a mobile computing device, a workstation, a minicomputer, a mainframe computer, a network node, or any other computing device. The computing system 900 can also include different bus architectures, network platforms, multi-processor platforms, etc. Various operating systems (OS) can be used, including UNIX, Linux, Windows, MacintoshOS, Palm OS, and other suitable operating systems.

[0099] Some of the above functions may be composed of instructions stored in a storage medium (e.g., a computer-readable medium or a processor-readable medium). The instructions may be retrieved and executed by a processor. Some examples of storage media are storage devices, tapes, disks, etc. When executed by a processor, the instructions are executable to direct the processor to operate in accordance with the present invention. Those skilled in the art are familiar with instructions, processors, and storage media.

[0100] It is noteworthy that any hardware platform suitable for performing the processing described in this article is suitable for use with the present invention. The term "computer-readable storage medium" used in this article refers to any medium that participates in providing instructions to the processor to perform. Such a medium can take many forms, including but not limited to non-volatile media, volatile media and transmission media. For example, non-volatile media include optical disks or disks, such as hard disks. Volatile media include dynamic memory, such as system random access memory (RAM). Transmission media include coaxial cables, copper wires and optical fibers, etc., including wires of an embodiment of the bus. Transmission media can also take the form of sound waves or light waves, such as those sound waves or light waves generated during radio frequency (RF) and infrared (IR) data communication. Common forms of computer-readable media include, for example, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD read-only memory (ROM) disks, DVDs, any other optical media, any other physical media with patterns of marks or holes, RAM, PROM, EPROM, EEPROM, any other memory chips or boxes, carriers, or any other media that a computer can read therefrom.

[0101] Various forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution. The bus carries the data to the system RAM, from which the processor retrieves and executes the instructions. Optionally, the instructions received by the system processor can be stored on a hard disk before or after execution by the processor.

[0102] Thus, a method and system for facial reproduction have been described. Although the embodiments have been described with reference to specific exemplary embodiments, it will be apparent that various variations and modifications can be made to those exemplary embodiments without departing from the broader spirit and scope of the present application. Accordingly, the description and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A method for facial reproduction, the method comprising: receiving, by a computing device, visual data including a visible portion of a source face; determining, by the computing device and based on the visible portion of the source face, a first portion of source facial parameters associated with a parameterized facial model, the first portion corresponding to the visible portion; predicting, by the computing device and based in part on the visible portion of the source face, a second portion of the source facial parameters, the second portion corresponding to a remaining portion of the source face; receiving, by the computing device, a target video, the target video including a target face; determining, by the computing device, target facial parameters associated with the parameterized facial model and corresponding to the target face based on the target video; and An output face is synthesized, by the computing device, based on the source facial parameters and the target facial parameters using the parameterized facial model, the output face comprising the source face that mimics a facial expression of the target face. 2 . The method according to claim 1 , further comprising replacing the target face in the target video with the output face to obtain an output video.

3. The method according to claim 1, wherein: The source facial parameters include at least one of a source facial expression, a source facial identity, and a source facial texture; The target facial parameters include at least one of a target facial expression, a target facial identity, and a target facial texture; and Synthesizing the output face includes replacing the target facial identity with the source facial identity and replacing the target facial texture with the source facial texture.

4. The method according to claim 1, wherein: The second part of the source facial parameters is predicted by a neural network.

5. The method according to claim 1, wherein: The second portion of the source face parameters corresponds to an eye region occluded in the source face.

6. The method according to claim 5, wherein: The eye area is obscured by the glasses.

7. The method according to claim 1, wherein: The second portion of the source facial parameters corresponds to a head region occluded in the source face.

8. The method according to claim 1, wherein: The visual data includes one or more videos featuring the visible portion of the source face.

9. The method according to claim 1, wherein: The visual data includes one or more photographs featuring the visible portion of the source face. 10 . The method of claim 9 , further comprising obtaining, by the computing device, the one or more photos from a social networking account of a user.

11. A computing device comprising: processor; as well as a memory storing instructions that, when executed by the processor, configure the computing device to: receiving visual data including a visible portion of a source face; determining, based on the visible portion of the source face, a first portion of source facial parameters associated with a parameterized facial model, the first portion corresponding to the visible portion; predicting a second portion of the source face parameters based in part on the visible portion of the source face, the second portion corresponding to a remaining portion of the source face; Receiving a target video, wherein the target video includes a target face; determining, based on the target video, target facial parameters associated with the parameterized facial model and corresponding to the target face; as well as An output face is synthesized using the parameterized facial model and based on the source facial parameters and the target facial parameters, the output face comprising the source face mimicking a facial expression of the target face.

12. The computing device of claim 11, wherein: The instructions also configure the computing device to replace the target face in the target video with the output face to obtain an output video.

13. The computing device of claim 11, wherein: The source facial parameters include at least one of a source facial expression, a source facial identity, and a source facial texture; The target facial parameters include at least one of a target facial expression, a target facial identity, and a target facial texture; and Synthesizing the output face includes replacing the target facial identity with the source facial identity and replacing the target facial texture with the source facial texture.

14. The computing device of claim 11, wherein: The second part of the source facial parameters is predicted by a neural network.

15. The computing device of claim 11, wherein: The second portion of the source face parameters corresponds to an eye region occluded in the source face.

16. The computing device of claim 15, wherein: The eye area is obscured by the glasses.

17. The computing device of claim 11, wherein: The second portion of the source facial parameters corresponds to a head region occluded in the source face.

18. The computing device of claim 11, wherein: The visual data includes one or more videos featuring the visible portion of the source face.

19. The computing device of claim 11, wherein: The visual data includes one or more photographs featuring the visible portion of the source face.

20. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computing device, cause the computing device to: receiving visual data including a visible portion of a source face; determining, based on the visible portion of the source face, a first portion of source facial parameters associated with a parameterized facial model, the first portion corresponding to the visible portion; predicting a second portion of the source face parameters based in part on the visible portion of the source face, the second portion corresponding to a remaining portion of the source face; Receiving a target video, wherein the target video includes a target face; determining, based on the target video, target facial parameters associated with the parameterized facial model and corresponding to the target face; as well as An output face is synthesized using the parameterized facial model and based on the source facial parameters and the target facial parameters, the output face comprising the source face mimicking a facial expression of the target face.

Images