Method, device and computer readable medium for generating facial key points of user in video to generate composite video

Abstract

A method may include identifying metadata of a source image including an appearance of a person or character, obtaining a target image including an appearance of a user, and generating metadata of the target image with suppressed artifacts. The metadata for the target image may include at least one of data for a head pose of the user or data for a facial expression of the user, and the artifact may originate from an object that occludes at least a portion of an appearance of the user in the target image or appears in the target image. The metadata for the source image and the metadata for the target image may be a basis for generating a composite image in which an appearance of a person or character mimics at least one of a facial expression or a head pose of a user.

Description

Technical Field

[0001] The present disclosure relates generally to video conferencing, image processing, and other systems. More specifically, the present disclosure relates to refinement of facial key point metadata generation for video conferencing or other applications. Background Art

[0002] Video conferencing is becoming increasingly popular for both business and personal use. One problem that often affects participants during a video conference is a lack of sufficient bandwidth, meaning that the participant's electronic device may not have sufficient bandwidth to (i) send outgoing video and audio data associated with the participant and (ii) receive incoming video and audio data associated with one or more other video conference participants. As a result, during a video conference session, participants often experience significant degradation of displayed video content and significant interruption of presented audio content. Summary of the invention

[0003] Technical Solution

[0004] The present disclosure relates to refinement of facial keypoint metadata generation for video conferencing or other applications.

[0005] In an embodiment of the present disclosure, a method may include identifying metadata of a source image that includes the appearance of a person or character. In an embodiment of the present disclosure, the method may include obtaining a target image that includes the appearance of a user. In an embodiment of the present disclosure, the method may include generating metadata of a target image in which artifacts are suppressed. In an embodiment of the present disclosure, the metadata of the target image may include at least one of data of a head pose of the user in the target image or data of a facial expression of the user in the target image. In an embodiment of the present disclosure, the artifact may originate from an object that obscures at least a portion of the user's appearance in the target image or appears in the target image. In an embodiment of the present disclosure, the metadata for the source image and the metadata for the target image may be the basis for generating a composite image in which the appearance of a person or character mimics at least one of the facial expression or head pose of the user.

[0006] In an embodiment of the present disclosure, a device may include at least one processor configured to identify metadata of a source image including the appearance of a person or character. In an embodiment of the present disclosure, the at least one processor may be configured to obtain a target image including the appearance of a user. In an embodiment of the present disclosure, the at least one processor may be configured to generate metadata of a target image with suppressed artifacts. In an embodiment of the present disclosure, the metadata of the target image may include at least one of data of a head posture of the user in the target image or data of a facial expression of the user in the target image. In an embodiment of the present disclosure, the artifact may originate from an object that obscures at least a portion of the user's appearance in the target image or appears in the target image. In an embodiment of the present disclosure, the metadata for the source image and the metadata for the target image may be the basis for generating a composite image in which the appearance of a person or character mimics at least one of the facial expression or head posture of the user.

[0007] In an embodiment of the present disclosure, a computer-readable medium may contain instructions that, when executed, cause at least one processor to identify metadata of a source image including the appearance of a person or character. In an embodiment of the present disclosure, the computer-readable medium may also contain instructions that, when executed, cause at least one processor to obtain a target image including the appearance of a user. In an embodiment of the present disclosure, a computer-readable medium may contain instructions that, when executed, cause at least one processor to generate metadata of a target image with suppressed artifacts. In an embodiment of the present disclosure, the metadata of the target image may include at least one of data of a head pose of the user in the target image or data of a facial expression of the user in the target image. In an embodiment of the present disclosure, the artifact may originate from an object that obscures at least a portion of the user's appearance in the target image or appears in the target image. In an embodiment of the present disclosure, the metadata for the source image and the metadata for the target image may be the basis for generating a synthetic image in which the appearance of a person or character mimics at least one of the facial expression or head pose of the user.

[0008] Other technical features may be apparent to those skilled in the art from the following drawings, descriptions and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, wherein like reference numerals represent like parts:

[0010] Figure 1 An example network configuration including electronic devices in an embodiment of the present disclosure is shown;

[0011] Figure 2 An example architecture for generating refined facial key point metadata for video conferencing or other applications in an embodiment of the present disclosure is shown;

[0012] Figures 3 to 5 It is shown that in the embodiments of the present disclosure, Figure 2 Example machine learning models used in the architecture;

[0013] Figure 6 The embodiment of the present disclosure is shown in FIG. Figure 2 Example merging of facial keypoint metadata in the architecture of

[0014] Figures 7 to 11 shows example results that may be obtained using refined facial key point metadata generation for video conferencing or other applications in embodiments of the present disclosure;

[0015] Fig.12 and 13 An example method for refined facial key point metadata generation and use in an embodiment of the present disclosure is shown; and

[0016] Fig.14 An example method performed by an electronic device is shown. DETAILED DESCRIPTION

[0017] Before proceeding to the following detailed description, it may be advantageous to set forth definitions of certain words and phrases used throughout this patent document. The terms "send," "receive," and "communicate," and their derivatives, encompass both direct and indirect communications. The terms "include," "comprise," and their derivatives, mean including, but not limited to. The term "or" is inclusive, meaning and / or. The phrase "associated with," and its derivatives, means including, included within, interconnected with, including, contained within, connected to or connected with, coupled to or coupled with, communicable with, collaborate with, interlace, juxtapose, be close to, bound to or bound with, have, have the nature of, have to or with, etc., a relationship, etc.

[0018] In addition, the various functions described below can be implemented or supported by one or more computer programs, each of which is formed by a computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data or a portion thereof that are suitable for implementation in a suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code and executable code. The phrase "computer-readable medium" includes any type of medium that can be accessed by a computer, such as a read-only memory (ROM), a random access memory (RAM), a hard drive, a compact disk (CD), a digital video disk (DVD) or any other type of memory. "Non-transitory" computer-readable media excludes wired, wireless, optical or other communication links that transmit temporary electrical signals or other signals. Non-transitory computer-readable media include media in which data can be permanently stored and media in which data can be stored and rewritten later, such as rewritable optical disks or erasable memory devices.

[0019] As used herein, terms and phrases such as "having", "may have", "include" or "may include" features (such as numbers, functions, operations or components such as parts) indicate the presence of features without excluding the presence of other features. In addition, as used herein, the phrases "A or B", "at least one of A and / or B" or "one or more of A and / or B" may include all possible combinations of A and B. For example, "A or B", "at least one of A and B" and "at least one of A or B" may indicate (1) including at least one A, (2) including at least one B or (3) including all of at least one A and at least one B. In addition, as used herein, the terms "first" and "second" may modify various components regardless of importance and do not limit the components. These terms are only used to distinguish one component from another. For example, a first user device and a second user device may indicate user devices that are different from each other, regardless of the order or importance of the devices. Without departing from the scope of the present disclosure, a first component may be represented as a second component, and vice versa.

[0020] It should be understood that when an element (such as a first element) is referred to as being (operably or communicatively) "coupled" / "coupled to" or "connected to" another element (such as a second element), it may be coupled to or connected to the other element directly or via a third element. In contrast, it will be understood that when an element (such as a first element) is referred to as being "directly coupled" or "directly connected to" another element (such as a second element), no other element (such as a third element) is interposed between the element and the other element.

[0021] As used herein, the phrase "configured (or set) to" may be used interchangeably with the phrases "suitable for," "capable of," "designed to," "adapted to," "manufactured to," or "capable of," as the case may be. The phrase "configured (or set) to" does not essentially mean "designed specifically in hardware." On the contrary, the phrase "configured to" may indicate that a device may perform an operation together with another device or part. For example, the phrase "a processor configured (or set) to perform A, B, and C" may indicate a general-purpose processor (such as a CPU or application processor) that can perform operations by executing one or more software programs stored in a memory device, or a dedicated processor (such as an embedded processor) for performing operations.

[0022] The terms and phrases used herein are only used to describe some embodiments of the present disclosure, rather than to limit the scope of other embodiments of the present disclosure. It should be understood that, unless the context clearly states otherwise, the singular forms "one", "an" and "the" include plural references. All terms and phrases used herein (including technical and scientific terms and phrases) have the same meaning as the meaning generally understood by those of ordinary skill in the art to which the embodiments of the present disclosure belong. It will be further understood that terms and phrases (such as those defined in a common dictionary) should be interpreted as having a meaning consistent with its meaning in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense, unless explicitly defined herein. In some cases, the terms and phrases defined herein may be interpreted as excluding embodiments of the present disclosure.

[0023] Examples of "electronic devices" according to embodiments of the present disclosure may include at least one of a smart phone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device (such as smart glasses, a head mounted device (HMD), electronic clothing, an electronic bracelet, an electronic necklace, an electronic accessory, an electronic tattoo, a smart mirror, or a smart watch). Other examples of electronic devices include smart home appliances. Examples of smart home appliances may include a television, a digital video disc (DVD) player, an audio player, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave, a washing machine, a dryer, an air purifier, a set-top box, a home automation control panel, a security control panel, a TV box (such as SAMSUNG HOMESYNC, APPLE TV, or GOOGLE TV), a smart speaker or speaker with an integrated digital assistant (such as SAMSUNG GALAXY HOME, APPLE HOMEPOD, or AMAZON ECHO), a game console (such as XBOX, PLAYSTATION, or NINTENDO), an electronic dictionary, an electronic key, a camera, or at least one of an electronic photo frame. Other examples of electronic devices include various medical devices (such as various portable medical measurement devices (such as blood sugar measurement devices, heart rate measurement devices, or body temperature measurement devices), magnetic resource angiography (MRA) devices, magnetic resource imaging (MRI) devices, computer tomography (CT) devices, imaging devices, or ultrasound devices), navigation devices, global positioning system (GPS) receivers, event data recorders (EDR), flight data recorders (FDR), car infotainment devices, navigation electronic devices (such as navigation navigation devices or gyrocompasses), avionics equipment, security equipment, vehicle-mounted head units, industrial or domestic robots, automatic teller machines (ATMs), point-of-sale (POS) devices, or Internet of Things (IoT) devices (such as light bulbs, various sensors, electricity or gas meters, sprinklers, fire alarms, thermostats, street lights, ovens, fitness equipment, hot water tanks, heaters, or boilers). Other examples of electronic devices include at least a part of a piece of furniture or a building / structure, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (such as devices for measuring water, electricity, gas, or electromagnetic waves). Note that according to various embodiments of the present disclosure, the electronic device may be one or a combination of the devices listed above. According to some embodiments of the present disclosure, the electronic device may be a flexible electronic device. The electronic devices disclosed herein are not limited to the devices listed above, and may include new electronic devices depending on technological developments.

[0024] In the following description, according to various embodiments of the present disclosure, an electronic device is described with reference to the accompanying drawings. As used herein, the term "user" may refer to a person using an electronic device or another device (such as an artificial intelligence electronic device).

[0025] Definitions for certain other words and phrases may be provided throughout this patent document. Those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.

[0026] Nothing in the description in this application should be read as implying that any particular element, step, or function is an essential element that must be included within the scope of the claims. The scope of the patented subject matter is limited solely by the claims. In addition, no claim is intended to invoke 35 U.S.C. §112(f) unless the exact words "means for..." are followed by a participle. The use of any other terms within the claims, including but not limited to "mechanism," "module," "device," "unit," "component," "element," "member," "device," "machine," "system," "processor," or "controller," are understood by applicant to refer to structures known to those skilled in the relevant art and are not intended to invoke 35 U.S.C. §112(f).

[0027] The following discussion is described with reference to the accompanying drawings. Figures 1 to 13 As well as one or more embodiments of the present disclosure. However, it should be understood that the present disclosure is not limited to these embodiments, and all changes and / or equivalents or replacements thereto also belong to the scope of the present disclosure. Throughout the specification and the drawings, the same or similar reference numerals may be used to refer to the same or similar elements.

[0028] As described above, video conferencing may become increasingly popular for both business and personal use. One problem that commonly affects participants during a video conference may be a lack of sufficient bandwidth, meaning that the participant's electronic device may not have sufficient bandwidth to (i) send outgoing video and audio data associated with the participant and (ii) receive incoming video and audio data associated with one or more other video conference participants. As a result, it may be common for participants to experience significant degradation of displayed video content and significant interruptions of presented audio content during a video conference session.

[0029] In order to reduce the amount of data sent to or from participants in a video conferencing session, (i) an image of the participant's face and body and (ii) a sequence of facial key points extracted from image frames in a video stream of the participant may be identified for at least one participant. The facial key points may identify how the participant's face changes in the image frames of the participant's video stream. This may allow devices of other participants to display an image of the participant's face and body and then change the displayed image of the participant's face and body based on the sequence of facial key points.

[0030] Unfortunately, video conference participants may routinely have their heads or parts of their bodies temporarily occluded or blocked, such as when a video conference participant makes a motion or gesture with their hands or arms that temporarily occludes their heads or bodies. It is also common for objects to appear near a participant's face or head or otherwise appear within the participant's video stream, such as when a participant moves or gestures with his or her hands or arms or when another person approaches the participant. When facial key points are identified independently for different image frames of a video stream, this may result in visible artifacts being created when the facial key points are subsequently used. This is because the presence of a participant's hands or arms, other people, or other objects may be incorrectly identified as facial key points, and using these incorrectly identified facial key points to reconstruct the video stream may create rendering artifacts or other artifacts.

[0031] The present disclosure provides a technique for refining the generation of facial key point metadata for video conferencing applications or other applications. As described in more detail below, a source image of a user and a driving video showing the movement of the user's face or head can be obtained. In some cases, the source image of the user can represent a source image of a video conference participant, and the driving video can represent a video stream of the video conference participant. Metadata can be generated that identifies an animation to be applied to the source image so that the source image mimics at least some of the movement of the user's face or head in the driving video. For example, the metadata can identify a rotation and translation of the user's head to be applied to the source image or an expression deformation to be applied to the user's face in the source image. Metadata can be generated for different image frames in the driving video to identify a sequence of rotation and translation or expression deformation to be applied to the user's face or head in the source image. The metadata can be sent to at least one other device, such as a device of at least one other video conference participant, which can be configured to animate the source image based on the metadata.

[0032] During the generation of metadata, one or more artifacts associated with temporarily occluding at least a portion of the user's head or body or one or more objects that temporarily appear in the driving video can be suppressed. For example, a trained machine learning model can be used to process metadata (such as metadata related to the rotation and translation of the user in the driving video or expression deformation) and generate refined metadata, wherein the trained machine learning model is trained to suppress the effects of temporarily occluding at least a portion of the user's head or body or one or more objects that temporarily appear in the driving video. This can help reduce or prevent the generation of artifacts caused by temporary occlusion of the user's head or body or artifacts caused by the presence of temporary objects. In some cases, the refinement of the metadata can also suppress finer details of the user's face or head movement, such as when the lip movement of the user's face is suppressed. To help overcome this problem, a merging mechanism can be used to merge at least some of the refined metadata with the original (unprocessed) metadata associated with the user's expression deformation. This can help at least partially restore the finer details of the user's face or head movement.

[0033] In this way, metadata related to facial key points or other aspects of the user's face can be identified in the driving video and used during video conferencing or other applications, which can significantly reduce the bandwidth required for video conference participants. In addition, the metadata can be refined so that any metadata related to the user's non-facial key points is suppressed, such as when one or more of the user's hands or arms may temporarily occlude at least a portion of the user's head or body or appear near the user's face or head, or when a person or other object appears near the user in the driving video. This helps to reduce noise in the metadata associated with facial key points and reduce or eliminate the generation of rendering artifacts when the source image is animated based on the refined metadata. In an example, the merging mechanism can help ensure that finer details of the user's face or head movement are retained in the metadata, which can result in improved video reconstruction results. In an example, various additional functions can be supported here, such as a function for keeping the animated source image looking forward even if the user turns his or her head, or a function for using an artificial image (such as a "beautified selfie") as a source image.

[0034] Figure 1 An example network configuration 100 including electronic devices according to the present disclosure is shown. Figure 1 The embodiment of network configuration 100 shown is for illustration only. One or more embodiments of network configuration 100 may be used without departing from the scope of the present disclosure.

[0035] According to one or more embodiments of the present disclosure, an electronic device 101 is included in a network configuration 100. The electronic device 101 may include at least one of a bus 110, a processor 120, a memory 130, an input / output (I / O) interface 150, a display 160, a communication interface 170, or a sensor 180. In one or more embodiments, the electronic device 101 may exclude at least one of these components, or may add at least one other component. The bus 110 may include circuits for connecting the components 120-180 to each other and for transmitting communications (such as control messages and / or data) between the components.

[0036] The processor 120 may include one or more processing devices, such as one or more microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). In one or more embodiments, the processor 120 may include one or more of a central processing unit (CPU), an application processor (AP), a communication processor (CP), or a graphics processor unit (GPU). The processor 120 is capable of controlling at least one of the other components of the electronic device 101 and / or performing operations or data processing related to communication or other functions. As described below, the processor 120 may be used to process image / video data in order to perform refined facial key point metadata generation for video conferencing or other applications. The processor 120 may also or alternatively be used to obtain source images and metadata, and animate the source images based on the metadata to support video conferencing or other applications.

[0037] The memory 130 may include volatile and / or non-volatile memory. For example, the memory 130 may store commands or data related to at least one other component of the electronic device 101. According to one or more embodiments of the present disclosure, the memory 130 may store software and / or programs 140. The programs 140 may include, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and / or an application program (or "application") 147. At least a portion of the kernel 141, the middleware 143, or the API 145 may be represented as an operating system (OS).

[0038] The kernel 141 may control or manage system resources (such as bus 110, processor 120, or memory 130) for executing operations or functions implemented in other programs (such as middleware 143, API 145, or application 147). The kernel 141 may provide an interface that allows the middleware 143, API 145, or application 147 to access various components of the electronic device 101 to control or manage system resources. The application 147 may include one or more applications for processing image / video data to perform refined facial key point metadata generation and / or for obtaining source images and metadata and animating source images based on metadata to support video conferencing or other applications. These functions may be performed by a single application or multiple applications, each of which performs one or more of these functions. For example, the middleware 143 may be used as a repeater to allow the API 145 or application 147 to communicate data with the kernel 141. Multiple applications 147 may be provided. The middleware 143 can control work requests received from the applications 147, such as by assigning a priority of using system resources (such as the bus 110, the processor 120, or the memory 130) of the electronic device 101 to at least one of the plurality of applications 147. The API 145 may be an interface that allows the application 147 to control functions provided from the kernel 141 or the middleware 143. For example, the API 145 may include at least one interface or function (such as a command) for archive control, window control, image processing, or text control.

[0039] The I / O interface 150 may serve as an interface that may, for example, transmit commands or data input from a user or other external devices to other components of the electronic device 101. The I / O interface 150 may also output commands or data received from other components of the electronic device 101 to the user or other external devices.

[0040] The display 160 may include, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot light emitting diode (QLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display. The display 160 may also be a depth perception display, such as a multi-focal display. The display 160 is capable of displaying, for example, various contents (such as text, images, videos, icons, or symbols) to the user. The display 160 may include a touch screen and may receive, for example, a touch, gesture, proximity, or hover input using an electronic pen or a body part of the user.

[0041] For example, the communication interface 170 can establish communication between the electronic device 101 and an external electronic device (such as the first electronic device 102, the second electronic device 104, or the server 106). For example, the communication interface 170 can be connected to the network 162 or 164 through wireless or wired communication to communicate with the external electronic device. The communication interface 170 can be a wired or wireless transceiver or any other component for sending and receiving signals (such as images).

[0042] The electronic device 101 may include one or more sensors 180, which can measure physical quantities or detect the activation state of the electronic device 101, and convert the measured or detected information into electrical signals. For example, one or more sensors 180 may include one or more cameras or other imaging sensors, which can be used to capture images of scenes. The sensor 180 may also include one or more buttons for touch input, one or more microphones, gesture sensors, gyroscopes or gyroscope sensors, air pressure sensors, magnetic sensors or magnetometers, acceleration sensors or accelerometers, grip sensors, proximity sensors, color sensors (such as RGB sensors), biophysical sensors, temperature sensors, humidity sensors, illumination sensors, ultraviolet (UV) sensors, electromyography (EMG) sensors, electroencephalography (EEG) sensors, electrocardiography (ECG) sensors, infrared (IR) sensors, ultrasonic sensors, iris sensors, or fingerprint sensors. The sensor 180 may include an inertial measurement unit, which may include one or more accelerometers, gyroscopes, and other components. In an example, (one or more) sensors 180 may include a control circuit for controlling at least one of the sensors included here. Any of these sensors 180 may be located within the electronic device 101.

[0043] The first external electronic device 102 or the second external electronic device 104 may be a wearable device or a wearable device (such as an HMD) in which the electronic device can be installed. When the electronic device 101 is installed in the electronic device 102 (such as an HMD), the electronic device 101 may communicate with the electronic device 102 through the communication interface 170. The electronic device 101 may be directly connected to the electronic device 102 to communicate with the electronic device 102 without involving a separate network. The electronic device 101 may also be an augmented reality wearable device including one or more cameras, such as glasses.

[0044] Wireless communication can use, for example, long term evolution (LTE), advanced long term evolution (LTE-A), fifth generation wireless system (5G), millimeter wave or 60GHz wireless communication, wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro) or global system for mobile communications (GSM) as at least one of the cellular communication protocol. Wired connection can include, for example, universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232) or plain old telephone service (POTS). Each network 162 or 164 includes at least one communication network, such as a computer network (such as a local area network (LAN) or a wide area network (WAN)), the Internet or a telephone network.

[0045] The first external electronic device 102 and the second external electronic device 104 and the server 106 may each be a device of the same or different type as the electronic device 101. According to one or more embodiments of the present disclosure, the server 106 includes a group of one or more servers. In addition, according to one or more embodiments of the present disclosure, all or some operations performed on the electronic device 101 may be performed on another one or more other electronic devices (such as the electronic devices 102 and 104 or the server 106). In addition, according to one or more embodiments of the present disclosure, when the electronic device 101 should automatically or upon request perform some functions or services, the electronic device 101 may request another device (such as the electronic devices 102 and 104 or the server 106) to perform at least some functions associated with it, rather than performing the functions or services separately or additionally. Other electronic devices (such as the electronic devices 102 and 104 or the server 106) are capable of performing the requested functions or additional functions and transmitting the results of the execution to the electronic device 101. The electronic device 101 may provide the requested functions or services by processing the received results as is or additionally. To this end, for example, cloud computing, distributed computing, or client-server computing technologies may be used. Although Figure 1 The electronic device 101 is shown to include a communication interface 170 that communicates with the external electronic device 104 or the server 106 via the network 162 or 164 , but according to one or more embodiments of the present disclosure, the electronic device 101 may operate independently without a separate communication function.

[0046] The server 106 may include components that are the same or similar to the electronic device 101 (or a suitable subset thereof). The server 106 may support driving the electronic device 101 by performing at least one of the operations (or functions) implemented on the electronic device 101. For example, the server 106 may include a processing module or processor that may support the processor 120 implemented in the electronic device 101. As described below, the server 106 may be used to process image / video data in order to perform refined facial key point metadata generation for video conferencing or other applications. The server 106 may also or alternatively be used to provide source images and metadata to one or more end-user devices (such as the electronic device 101) for animating the source images based on the metadata in order to support video conferencing or other applications.

[0047] although Figure 1 An example of a network configuration 100 including an electronic device 101 is shown, but the Figure 1 Various changes may be made. For example, network configuration 100 may include any number of each component in any suitable arrangement. In general, computing and communication systems have a wide variety of configurations, and Figure 1 The scope of the present disclosure is not limited to any particular configuration. Figure 1 One operating environment is shown in which the various features disclosed in this patent document may be used, but these features may be used in any other suitable system.

[0048] Figure 2 An example architecture 200 for generating refined facial key point metadata for video conferencing or other applications according to the present disclosure is shown. For ease of explanation, Figure 2 The illustrated architecture 200 is described as Figure 1 The network configuration 100 is implemented on or supported by one or more electronic devices. Figure 2 The architecture 200 shown in FIG. 2 may be used with any other suitable device and may be used in any other suitable system.

[0049] like Figure 2 As shown, architecture 200 may generally involve the use of a sending device 202 and a receiving device 204. Sending device 202 may represent an electronic device used by a user to provide video data (and optionally other data such as audio data), which is provided directly or indirectly to receiving device 204. Each of sending device 202 and receiving device 204 may represent any suitable electronic device, such as Figure 1 Note that although device 202 is referred to as a "sending" device and device 204 is referred to as a "receiving" device, this is only in reference to Figure 2204. In many cases, devices 202 and 204 may include similar components, so each device 202 or 204 may be able to send information to and receive information from the other device 204 or 202. In addition, note that although two devices 202 and 204 are shown here, more than two devices may be used in video conferencing applications or other applications. As a result, the sending device 202 can send information to any suitable number of receiving devices, and the receiving device 204 can receive information from any suitable number of sending devices.

[0050] In this example, the sending device 202 may obtain a source image 206 and a driving video 208 including a plurality of image frames 210. The source image 206 may represent an image associated with a user (or person, character) of the sending device 202. In some cases, the source image 206 may represent an actual image of the user of the sending device 202. For example, the source image 206 may be captured using the sending device 202, such as when the sending device 202 includes one or more imaging sensors 180. The source image 206 may represent a current image, a recent image, or an older image of the user of the sending device 202. In one or more cases, the source image 206 may represent an artificial image of the user of the sending device 202. For example, the artificial image may represent an actual image of the user that has undergone one or more image processing operations to change the appearance of the user, such as when the artificial image represents a "beautified selfie" of the user. In general, the source image 206 may include any suitable image of the user of the sending device 202 or representing the user of the sending device 202.

[0051] The driving video 208 may represent a series of image frames 210 capturing a user of the sending device 202, such as image frames 210 capturing a user speaking or listening during a video conference. The driving video 208 may generally capture all or most of the face and head of the user of the sending device 202, typically along with a portion of the user's body (such as the user's shoulders and upper torso). The driving video 208 may capture the user's movements, such as movements of the user's face or head. This may include movements that reposition or rotate the user's head within the image frames 210 and expression deformations where the user's face deforms when speaking or making expressions (such as emotional expressions on the user's face). In some cases, the driving video 208 may represent an image of the user captured using the sending device 202, such as when the sending device 202 includes one or more imaging sensors 180. In general, the driving video 208 may include any suitable video of or representing a user of the sending device 202. The driving video 208 may include any suitable number of image frames 210, which may vary based on (among other things) the frame rate of video capture.

[0052] One or more source image features (i.e., metadata) may be extracted from the source image 206. In an embodiment of the present disclosure, the sending device 202 may extract appearance features from the source image 206. In an example, the sending device 202 may extract 2D (2-dimensional) appearance features from the source image 206 using a 2D appearance feature extraction network to map the source image 206 to a 2D feature map. In an example, the sending device 202 may extract 3D (3-dimensional) appearance features from the source image 206 using a 3D appearance feature extraction network to map the source image 206 to a 3D appearance feature volume (i.e., three spatial dimensions: width, height, and depth).

[0053] In an embodiment of the present invention, the sending device 202 can use a canonical 3D keypoint detection network to extract a set of K (e.g., 20) 3D canonical keypoints (i.e., facial keypoints 218) from the source image 206. The canonical keypoints can be unsupervised learning and different from common facial landmarks. The canonical keypoints can mean posture and expression independent of the face. The canonical keypoints can encode a person's geometric signature in neutral posture and expression.

[0054] In an embodiment of the present disclosure, the sending device 202 can extract posture and expression information from the source image 206. In an example, the sending device 202 can use a head posture estimation network to estimate the head posture of a person (or character, user) in the source image 206. The head posture of the person can be estimated by the rotation data of the source image (e.g., matrix) and the translation data of the source image (e.g. Vector). In an example, the sending device 202 may use an expression deformation estimation network to estimate expression deformation data of the source image. The expression deformation data may be data of facial expressions of a person (or character, user) in the source image 206. The expression deformation data may include a set of K 3D expression deformations (i.e., deformations of key points from neural expressions). In an example, the head pose estimation network and the expression deformation estimation network may extract motion-related geometric information in the image.

[0055] In an embodiment of the present disclosure, the sending device 202 may combine identity-specific information (e.g., canonical facial key points) with motion-related information (e.g., rotation data, translation data, expression deformation data) to obtain source facial key points 228 (i.e., facial key points of a person (or character, user) in the source image 206). In an example, the sending device 202 may use a transformation (T) to obtain the source facial key points:

[0056] x s,k =T(x c,k ,R s ,t s ,δ sk )=R s xc,k +t s +δ s，k

[0057] Here, x s，k is the kth source facial keypoint 228 of the person in the source image 206, x c，k is the kth canonical facial keypoint of the person, R s is the rotation data of the source image, t s is the translation data of the source image, and δ s，k is the expression deformation data corresponding to the kth facial key point. The final facial key point may be image specific and contain character signature, posture and expression information. At least one of the appearance features of the source image, the canonical facial key point, the rotation data of the source image, the translation data of the source image, or the expression deformation data of the source image may be included in the metadata of the source image.

[0058] refer to Figure 2, the source image 206 may be provided to a facial key point estimation function 212, a head pose estimation function 214, and an expression deformation estimation function 216. In embodiments of the present disclosure, the facial key point estimation function 212 may generally operate to process the source image 206 and identify facial key points 218 associated with the face of the user in the source image 206. The facial key points 218 are referred to as representing "canonical" key points and identify specific locations of various features associated with the face of the user. The number and location of the facial key points 218 identified here may vary depending on the implementation and the source image 206. In some cases, the facial key points 218 may identify the locations of various features associated with the face of the user, such as the location of the user's eyes, nose, mouth, cheeks, chin, forehead, and neck (or any combination thereof). The facial key point estimation function 212 may use any suitable image processing technique or other technique to identify specific features of the user's face in the source image 206, such as by performing object identification or object recognition to identify specific features of the user's face and the location of specific features within the source image 206. The head pose estimation function 214 may generally operate to process the source image 206 and estimate rotation and translation parameters 220 associated with the face or head of the user within the source image 206. The translation of the face or head of the user within the source image 206 may be defined by the position of the user's head left-right and up-down within the source image 206. For example, the face or head of the user may be perfectly centered within the source image 206, or may be positioned to the left or right and / or above or below a center point of the source image 206. In some cases, the translation of the face or head of the user within the source image 206 may be represented as a translational deformation of the face or head of the user within the source image 206. The rotation of the face or head of the user within the source image 206 may be defined by the direction in which the face or head of the user is pointing within the source image 206. For example, the face or head of the user may be pointing directly forward in the source image 206, or the user may have rotated his or her head around one or more axes within the source image 206. In some cases, the rotation of the user's face or head within source image 206 may be expressed using the pitch, yaw, and roll angles of the user's face or head within source image 206. Head pose estimation function 214 may use any suitable image processing or other techniques to identify the translation and rotation of the user's face or head in source image 206, such as by using a head orientation detection algorithm.

[0059] The expression deformation estimation function 216 may generally operate to process the source image 206 and estimate one or more expression deformations 222 associated with the user's face within the source image 206. Expression deformations may be defined as deformations or changes in the shape of the user's face or the positions of features of the user's face when the user speaks or displays an emotion. For example, the shape of the user's face and the positions of various features of the user's face may change depending on whether the user is smiling, frowning, shocked / surprised, afraid, etc. The expression deformation estimation function 216 may thus identify one or more expression deformations 222 contained in the source image 206, wherein the expression deformations 222 may define the user's expression as contained in the source image 206. The expression deformation estimation function 216 may use any suitable image processing technique or other technique to identify expression deformations of the user's face in the source image 206, such as by using pattern recognition techniques.

[0060] Collectively, the rotation and translation parameters 220 (i.e., rotation data and translation data) and the expression deformation 222 (i.e., expression deformation data) determined for the source image 206 may represent a set of metadata 224. The set of metadata 224 may effectively describe the pose of the user's head and the expression deformation of the user's face as contained in the source image 206. The facial key points 218 and the set of metadata 224 may be provided to a transformation function 226, which transforms the facial key points 218 based on the rotation and translation parameters 220 and the expression deformation 222. This may transform the canonical facial key points 218 into source facial key points 228. The source facial key points 228 may represent the facial key points of the user's face or head in the source image 206 without any translation / rotation and without a specific emotional expression. The transformation function 226 may use any suitable technique to transform the canonical facial key points 218 based on the rotation and translation parameters 220 and the expression deformation 222, such as a three-dimensional (3D) projection or other 3D transformation.

[0061] In an embodiment of the present disclosure, the sending device 202 may extract motion-related information from each image frame of the driving video 208. The multiple image frames of the driving video 208 may include a target image. In an example, the sending device 202 may extract the head pose of the user (or person) in the target image from the target image using a head pose estimation network. In an example, the head pose may be parameterized by the (original) rotation data and (original) translation data of the target image. In an example, the sending device 202 may extract the (original) expression deformation data for the target image from the target image using an expression deformation estimation network. The expression deformation data for the target image may be data for the facial expression of the user in the target image. The (original) rotation data, the (original) translation data, and the (original) expression deformation data may be included in the original metadata for the target image.

[0062] In an embodiment of the present invention, the sending device 202 may refine the original metadata of the target image using the refinement model 240 to generate refined metadata of the target image. The refined metadata for the target image may include at least one of refined rotation data, refined translation data, or refined expression deformation data. The sending device 202 may merge the refined metadata with at least some of the original metadata (i.e., at least some of the original expression deformation data) to generate merged metadata of the target image. In an example, the merged metadata may include some original metadata and some refined metadata. In an example, the merged metadata may be calculated based on at least some of the original metadata and at least some of the refined metadata.

[0063] In an embodiment of the present disclosure, the sending device 202 may generate streaming facial key points 250 based on the facial key points 218 (i.e., the canonical facial key points) and the merged metadata for driving the video. In an example, the sending device 202 may generate target facial key points based on the canonical facial key points and the merged metadata for the target image. In an example, the sending device 202 may generate the target facial key points by transforming the canonical facial key points based on the merged metadata using a transform (T):

[0064] x d,k =T(x c,k ,R d ,t d ,δ d,k )=R a x c,k +t a +δ d,k

[0065] Here, x d，k is the kth target facial keypoint of the user in the target image, x c，k is the kth canonical facial keypoint of the person in the source image, R d is the rotation data included in the merged metadata of the target image, t d is the translation data included in the merged metadata of the target image, and δ d，k is the expression deformation data included in the merged metadata of the target image, which corresponds to the kth canonical facial keypoint. One or more operations performed on the target image may be performed on each image frame of the driving video.

[0066] In an embodiment of the present disclosure, the head posture of a person in a composite video (or composite image) may be determined based on user input or settings. u and t u(This may be user-specified rotation data or translation data) when the transmitting device 202 can use the transformation (T) based on R d ←R u R d and t d ←t u +t d Transform the canonical facial keypoints to generate the target facial keypoints.

[0067] The driving video 208 may be provided to a head pose estimation function 230 and an expression deformation estimation function 232. The head pose estimation function 230 may operate in the same or similar manner as the head pose estimation function 214 described above, and the expression deformation estimation function 232 may operate in the same or similar manner as the expression deformation estimation function 216 described above. In fact, in some cases, the head pose estimation function 214 and the head pose estimation function 230 may represent the same function, such as when head pose estimation is performed on the source image 206 and then the head pose estimation is performed on the driving video 208 using the same function. Similarly, in some cases, the expression deformation estimation function 216 and the expression deformation estimation function 232 may represent the same function, such as when expression deformation estimation is performed on the source image 206 and then the driving video 208 using the same function. However, here, the head pose estimation function 230 and the expression deformation estimation function 232 may process multiple images, i.e., multiple image frames 210 contained in the driving video 208. Thus, the head pose estimation function 230 may be used to generate rotation and translation parameters 234 associated with the face or head of the user within each image frame 210, and the expression deformation estimation function 232 may be used to estimate one or more expression deformations 236 associated with the face of the user within each image frame 210. Collectively, the rotation and translation parameters 234 and expression deformations 236 determined for each image frame 210 may represent a set of metadata 238, and there may be a set of metadata 238 for each image frame 210 included in the driving video 208.

[0068] In this example, the set of metadata 238 may be provided to a refinement model 240. The refinement model 240 may represent one or more trained machine learning models that have been trained to refine one or more sets of subsequent metadata 238 based on the set of one or more previous metadata 238. For example, the set of one or more initial metadata 238 may contain metadata associated only with the face or head of a user within one or more initial image frames 210. However, if at some point the user moves his or her hand(s) or arm(s) into one or more subsequent image frames 210, this may result in the creation of rendering artifacts or other artifacts unless corrected. Similar problems may arise when other objects appear in subsequent image frames 210 or occlude portions of the user's head or body. The refinement model 240 may provide corrections here by refining the set of one or more subsequent metadata 238 for one or more subsequent image frames 210 that contain interfering objects. For example, the refinement model 240 may adjust metadata in the set of one or more subsequent metadata 238 based on a set of one or more previous metadata 238 for the image frame 210 that does not contain an object, such as a set of one or more previous metadata 238 associated with one or more image frames 210 immediately preceding the image frame 210 that contains an interfering object. The refinement model 240 here may be effectively trained to obtain temporal information in the form of noise in the set of metadata 238 and reduce or eliminate noise in the metadata 238.

[0069] As a specific example of this functionality, a user captured in the driving video 208 may be touching his or her face with his or her hand in one or more image frames 210. The presence of the user's hand in those image frames 210 may cause the set of metadata 238 generated for those image frames 210 to be noisy. Without correction, this noise may result in the creation of rendering artifacts or other artifacts. However, the refined model 240 herein may refine the set of metadata 238 for one or more image frames 210 containing the user's hand, such as by using the set of one or more metadata 238 from one or more previous image frames 210 that did not capture the user's hand.

[0070] The refinement model 240 may use any suitable technique to refine the set of metadata 238 associated with the image frame 210 driving the video 208. For example, as described above, the refinement model 240 may represent one or more trained machine learning models, and each trained machine learning model may use any number of machine learning model architectures. As an example, in one or more embodiments of the present disclosure, the refinement model 240 may include one or more recursive neural networks. In one or more embodiments of the present disclosure, the refinement model 240 may include a plurality of recursive neural networks, such as a recursive neural network trained to receive raw pitch, yaw, and roll angles and output refined pitch, yaw, and roll angles; a second recursive neural network trained to receive raw translation deformations (e.g., translation data) and output refined translation deformations; and a third recursive neural network trained to receive raw expression deformations (e.g., data of facial expressions) and output refined expression deformations. In one or more embodiments of the present disclosure, each recursive neural network may be implemented using a recursive neural network based on a gated recursive unit (GRU). However, the refinement model 240 may be implemented in any other suitable manner.

[0071] like Figure 2 As shown, refinement model 240 may output a set of refined metadata 242, such as a set of refined metadata 242 for each image frame 210. Depending on the content of image frame 210, each set of refined metadata 242 may match an associated set of original metadata 238 (such as when no occluding objects or other interfering objects are in the corresponding image frame 210) or include modified metadata based on an associated set of original metadata 238 (such as when at least one occluding object or other interfering object is in the corresponding image frame 210).

[0072] When the refinement model 240 is implemented using one or more machine learning models, the one or more machine learning models may be trained in any suitable manner. For example, in some cases, training data and real data may be obtained. The training data may include a driving video that includes objects that occlude portions of the user's head or body or otherwise appear in the driving video, and the real data may include the correct metadata to be generated using the training data. The training data may be provided to the head pose estimation function 230 and the expression deformation estimation function 232 to generate rotation and translation parameters 234 and expression deformations 236. The rotation and translation parameters 234 and the expression deformations 236 may be provided to the one or more machine learning models as a set of metadata 238, and the one or more machine learning models may be used to generate a set of refined metadata 242 associated with the set of original metadata 238. The set of refined metadata 242 may be compared with the real data, and differences or errors between the set of refined metadata 242 and the real data may be identified and used to calculate the overall error or loss of the one or more machine learning models. If the calculated loss exceeds a threshold, parameters of one or more machine learning models can be adjusted, and the metadata 238 can be processed again using the updated machine learning model (or new metadata 238 for different training data can be processed) to produce a new set of refined metadata 242 that can be compared to the real data to calculate the updated loss. Ideally, the calculated loss decreases over time and eventually falls below the threshold, indicating that the one or more machine learning models have been trained to accurately (at least within the desired accuracy represented by the threshold) generate a set of refined metadata 242 even in the presence of objects occluding portions of the user's head or body or appearing in the driving video. Below is a description of the process of generating a new set of refined metadata 242 that can be compared to the real data to calculate the updated loss. Figures 3 to 5 Describes a specific example technique for training a machine learning model for a specific example type.

[0073] like Figure 2 As shown, the set of refinement metadata 242 and the facial key points 218 may be provided to a transformation function 244, which may operate in the same or similar manner as the above-described transformation function 226. In some cases, the transformation function 226 and the transformation function 244 may represent the same function, such as when the facial key points 218 are transformed using the metadata 224 associated with the source image 206 and then the facial key points 218 are transformed using the metadata 238 associated with the driving video 208.

[0074] Because the refined model 240 can adjust the metadata 238, the architecture 200 can be robust to occluding objects or other objects that may interfere with the effective identification of metadata 238 related to facial landmarks. However, in some cases, the operation of the refined model 240 may negatively affect the finer details of the user's face or head movement. For example, the user's lip movement may be suppressed in the set of refined metadata 242. To help compensate for this problem or other similar problems, a merge function 246 can be used to merge or combine the original (unprocessed) expression deformation 236 with at least some of the refined metadata 242 to produce merged metadata 248. For example, the merge function 246 can replace some of the expression deformation information contained in one or more sets of refined metadata 242 with some of the information contained in the original expression deformation 236. As a specific example, expression deformation information related to facial landmarks at or around the user's mouth in one or more sets of refined metadata 242 can be replaced by expression deformation information related to the same facial landmarks in the original expression deformation 236. This can allow the merged metadata 248 to retain information related to finer movements of the user's face.

[0075] Here, the transformation function 244 can transform the facial key points 218 using the refined rotation / translation information in the set of refined metadata 242 and the expression deformation information contained in the merged metadata 248. The expression deformation information contained in the merged metadata 248 can include both (i) the refined expression deformation information in the set of refined metadata 242 and (ii) the expression deformation information in the original expression deformation 236. The transformation function 244 can generally operate to convert the canonical facial key points 218 into multiple sets of streaming facial key points 250, such as one set of streaming facial key points 250 for each image frame 210. Each set of streaming facial key points 250 can represent facial key points of the user's face or head in one of the image frames 210.

[0076] Source image 206, source facial key points 228, and a set of streaming facial key points 250 may be sent directly or indirectly (such as via server 106) to receiving device 204. In one or more embodiments of the present disclosure, source image 206 and source facial key points 228 may be sent from sending device 202 to receiving device 204 only once, such as before or at the beginning of a video conferencing session. As described above, source image 206 may be captured, generated, or otherwise obtained immediately before the video conferencing session, or may be captured, generated, or otherwise obtained at some time before the video conferencing session (possibly long before the video conferencing session). A set of streaming facial key points 250 may be sent from sending device 202 to receiving device 204 while the video conferencing session is ongoing.

[0077] In this example, the receiving device 204 implements or otherwise accesses a video rendering model 252. The video rendering model 252 may receive as input the source image 206 (or metadata of the source image, appearance features of the source image), the source facial key points 228, and the set of streaming facial key points 250 and generate a synthesized output video 254. The synthesized output video 254 may include an animation applied to the source image 206 such that the source image 206 mimics at least some movement of the face or head of the user in the driving video 208. As a result, the receiving device 204 is able to generate the synthesized output video 254 such that the synthesized output video 254 appears to be actual video from the sending device 202 (at least to some desired degree of accuracy) without requiring or using bandwidth to receive the entire driving video 208.

[0078] In one or more embodiments of the present disclosure, the video rendering model 252 may represent one or more trained machine learning models that have been trained to generate a synthetic output video based on a source image, a source facial key point, and a set of streaming facial key points. When implemented using machine learning, one or more trained machine learning models may be trained in a manner similar to that discussed above. Here, the training data may include training source images, training source facial key points, and a training set of streaming facial key points, and the real data may include a known synthetic output video to be generated by the machine learning model. However, note that the receiving device 204 may use the source image 206, the source facial key point 228, and the set of streaming facial key points 250 to generate the synthetic output video 254 in any other suitable manner.

[0079] It should be noted that Figure 2 shown in or about Figure 2 The described functions may be implemented in the electronic device 101, the server 106, or other devices in any suitable manner. For example, in one or more embodiments of the present disclosure, one or more software applications or other software instructions executed by the processor 120 of the electronic device 101, the server 106, or other devices may be used to implement or support Figure 2 shown in or relative to Figure 2 In one or more embodiments of the present disclosure, dedicated hardware components may be used to implement or support Figure 2 shown in or about Figure 2 At least some of the functions described. In general, any suitable hardware or any suitable combination of hardware and software / firmware instructions may be used to perform Figure 2 shown in or about Figure 2In addition, because the sending device 202 can also receive data from the receiving device 204 (such as another source image, other source facial key points, and other sets of streaming facial key points), the sending device 202 can include the video rendering model 252, and the receiving device 204 can include various components of the sending device 202.

[0080] although Figure 2 An example of an architecture 200 for refined facial key point metadata generation for video conferencing or other applications is shown, but may be used for Figure 2 Make various changes. For example, Figure 2 The various components and functions in the can be combined, further subdivided, duplicated, or rearranged according to specific needs. In addition, one or more additional components and functions can be included if needed or desired. In addition, architecture 200 can represent an example environment in which the refinement of facial key point metadata generation for video conferencing or other applications can be used. However, the functionality can be used in any other suitable environment.

[0081] In the examples, the above description generally assumes that the sending device 202 is an electronic device of one user (such as an instance of the electronic device 101), and the receiving device 204 is an electronic device of another user (such as another instance of the electronic device 101). In one or more embodiments of the present disclosure, each instance of the electronic device 101 can represent any suitable end-user electronic device, such as a smart phone, a tablet computer, a laptop computer, a desktop computer, smart glasses, an extended reality (XR) headset (including a virtual reality, an augmented reality, or a mixed reality headset), or any other suitable electronic device (including those described above). However, the sending device 202 and the receiving device 204 can each represent any other suitable device or system, including non-end-user devices. For example, the sending device 202 can represent a server (such as the server 106) that receives a source image 206 and a driving video 208 from one electronic device 101 or other device and provides a source image 206, a source facial key point 228, and a set of streaming facial key points 250 to another electronic device 101 or other device. Generally, the present disclosure is not limited to any particular physical implementation of the sending device 202 or the receiving device 204.

[0082] although Figure 2The sending device 202 is shown to perform processing operations on both the source image 206 and the target image (or the driving video 208), but is not limited thereto. In an embodiment of the present disclosure, some of the processing operations may be performed by the sending device 202, and other processing operations in the processing operations may be performed by another device. For example, another device may perform processing operations on the source image to generate metadata of the source image 206, and provide the metadata of the source image to the sending device 202. The sending device 202 may perform processing operations for the target image based on the received metadata of the source image.

[0083] Figures 3 to 5 It is shown that according to the present disclosure, Figure 2 Example machine learning models 300, 400, 500 used in the architecture 200 of FIG. More specifically, Figures 3 to 5 Explain how it can be implemented Figure 2 Specific examples of various functions of the refinement model 240 in the architecture 200. However, it should be noted that Figure 2 Refinement model 240 in architecture 200 may be implemented in any other suitable manner.

[0084] like Figure 3 As shown, a machine learning model 300 may be provided for identifying how to refine the rotation identified by the head pose estimation function 230. In this example, it is assumed that the head pose estimation function 230 outputs three angles for each image frame 210, namely, a pitch angle, a yaw angle, and a roll angle. The pitch angle may identify the angle by which the user has rolled his or her head up or down (if any), wherein the rotation may be considered to occur about an axis extending through the user's head from side to side. The yaw angle may identify the angle by which the user has turned his or her head left or right (if any), wherein the rotation may be considered to occur about an axis extending through the user's head from top to bottom. The roll angle may identify the angle by which the user has rolled his or her head clockwise or counterclockwise (if any), wherein the rotation may be considered to occur about an axis extending through the user's head from front to back. Figure 3 , the raw (unprocessed) rotation angle from the head pose estimation function 230 for the image frame 210 over time can be represented as Pitch 0 To Pitch t+1 , Yaw 0 To Yaw t+1 and Roll 0 To Roll t+1 The subscripts here may be associated with different image frames 210 received by the sending device 202 .

[0085] The machine learning model 300 here can use one or more GRU-based recurrent neural networks. For example, over time, rotating GRU gates 302a-302u can be used to process the pitch angle Pitch 0 To Pitch t+1 , the rotating GRU gates 304a-304u can be used to process the yaw angle Yaw 0 To Yaw t+1 , and the rotating GRU gates 306a-306u can be used to process the roll angle Roll 0 To Roll t+1 . In some cases, the same set of GRU gates can be used to process the pitch angle, yaw angle, and roll angle. In one or more cases, different sets of GRU gates can be used to process the pitch angle, yaw angle, and roll angle. In each set of GRU gates, the GRU gates can process the input angle and produce a refined output angle. For example, (one or more sets of) GRU gates can produce a refined pitch angle Pitch 0 'To Pitch t+1 ', refined yaw angle Yaw 0 'To Yaw t+1 'And the refined roll angle Roll 0 'To Roll t+1 '. Each group of GRU gates can also pass information from one GRU gate to the next GRU gate along the group, where the information passed between GRU gates is often referred to as a hidden state of the recurrent neural network. In this type of configuration, if occlusion or nearby objects interfere with the correct determination of the rotation of the user's head pose at one or more specific time steps (for one or more image frames 210), information from one or more previous time steps (for one or more previous image frames 210) can be used to correct the noisy rotation in subsequent time steps. Ideally, the refined angles output by the machine learning model 300 can accurately reflect the rotational movement of the user's head by removing the effects of occlusion or nearby objects.

[0086] like Figure 4 As shown, a machine learning model 400 may be provided for identifying how to refine the translation identified by the head pose estimation function 230. The machine learning model 400 may have a similar structure to the machine learning model 300 and include translation GRU gates 402a-402u. In this example, it is assumed that the head pose estimation function 230 outputs a time translation (TSL) value of the image frame 210, wherein the translation value identifies a distance or other translation deformation based on the movement of the user's face or head from a specified location (such as a center pixel) within the image frame 210. Figure 4In the example, the raw (unprocessed) translation values ​​from the head pose estimation function 230 for the image frame 210 over time can be represented as TSL 0 To TSL t+1 Again, the subscripts here may be associated with different image frames 210 received by the sending device 202 .

[0087] The translation GRU gates 402a-402u can be used to process translation values ​​over time. In the set of GRU gates 402a-402u, the GRU gate can process the translation value TSL 0 To TSL t+1 And generate the refined translation value TSL 0 'To TSL t+1 '. The group of gates 402a-402u can also pass information from one GRU gate to the next GRU gate along the group. In this type of configuration, if an occlusion or nearby object interferes with the correct determination of the translation of the user's head pose at one or more specific time steps, the information from one or more previous time steps can be used to correct the noisy translation in subsequent time steps. Ideally, the refined translation output by the machine learning model 400 can accurately reflect the translational motion of the user's head by removing the effects of occlusion or nearby objects.

[0088] like Figure 5 As shown, a machine learning model 500 may be provided for identifying how to refine the expression deformations identified by the expression deformation estimation function 232. The machine learning model 500 may have a similar structure to the machine learning models 300 and 400 and include expression deformation GRU gates 502a-502u. In the example, it is assumed that the expression deformation estimation function 232 outputs expression deformation (ExpDef) values ​​for the image frame 210 over time, wherein the expression deformation values ​​identify different expressions or deformation types in the user's face within the image frame 210. Figure 5 , the original (unprocessed) expression deformation from the expression deformation estimation function 232 for the image frame 210 over time is represented as ExpDef 0 To ExpDef t+1 Again, the subscripts here may be associated with different image frames 210 received by the sending device 202 .

[0089] Expression deformation GRU gates 502a-502u can be used to process expression deformation over time. In the set of GRU gates 502a-502u, the GRU gate can process expression deformation ExpDef 0 To ExpDef t+1 And generate refined expression deformation ExpDef 0 'To ExpDef t+1'. The group of gates 502a-502u can also pass information from one GRU gate to the next GRU gate along the group. In this type of configuration, if occlusion or nearby objects interfere with the correct determination of the expression deformation of the user's face at one or more specific time steps, the information from one or more previous time steps can be used to correct the noisy expression deformation in (multiple) subsequent time steps. Ideally, the refined expression deformation output by the machine learning model 500 can accurately reflect the expression deformation of the user's face by removing the influence of occlusion or nearby objects.

[0090] In one or more embodiments of the present disclosure, each of the GRU-based machine learning models 300, 400, 500 can be trained using training data (such as noise rotation angles, noise translations, or noise expression deformations) and real data (such as desired "denoised" rotation angles, translations, or expression deformations). The machine learning models 300, 400, 500 can be trained by minimizing the loss function between the real data and the refined rotation angles, refined translations, and refined expression deformations generated by the machine learning models 300, 400, 500. In one or more embodiments of the present disclosure, each loss function may include an adversarial loss term and a reconstruction loss term.

[0091] although Figures 3 to 5 It shows that it can be Figure 2 Examples of machine learning models 300, 400, 500 used in the architecture 200 of FIG. 1 , but may be used for Figures 3 to 5 Various changes may be made. For example, each of the machine learning models 300, 400, 500 may include any suitable number of GRU gates. In addition, the refinement model 240 may be implemented using any other suitable machine learning model architecture or multiple architectures, and is not limited to using a GRU-based recursive neural network.

[0092] Figure 6 The diagram shows the Figure 2 Example merging of facial key point metadata in the architecture 200 of FIG. More specifically, Figure 6 248. A specific example of how the merge function 246 may be used to merge or combine the original expression deformation 236 with the refined metadata 242 to produce the merged metadata 248 is illustrated. Note, however, that Figure 2 The merge function 246 in the architecture 200 may be implemented in any other suitable manner.

[0093] like Figure 6As shown, image frame 600 may capture a scene including a user, whose face is partially obscured here for privacy. The user's face may be associated with a plurality of facial landmarks 602 and 604. Expression deformation estimation function 232 may generate expression deformations (the above ExpDef values) for each facial landmark 602 and 604 in image frame 210, and refinement model 240 may output refined expression deformations (the above ExpDef' values) for each facial landmark 602 and 604 in image frame 210. Many of the facial landmarks (i.e., facial landmark 602 shown here) may be located at positions where movement of the user's lips has little effect on the user's face at those positions.

[0094] Other facial key points in the facial key points, namely the facial key points 604 shown here, are located in positions where the movement of the user's lips may have a much greater effect on the user's face at those positions. These facial key points 604 may represent the main or only facial key points used by the video rendering model 252 to animate the user's lips as contained in the source image 608 when generating the synthesized output video 254. Due to the operation of the refinement model 240, the refined expression deformations of the facial key points 604 generated by the refinement model 240 may not be very accurate. As a result, the merging function 246 can replace the refined expression deformations of the facial key points 604 generated by the refinement model 240 with the original expression deformations of those facial key points 604 generated by the expression deformation estimation function 232. This can help restore the finer movements of the user's lips in one or more of the sets of streaming facial key points 250, which can result in a more accurate rendering of the user's face in the synthesized output video 254.

[0095] although Figure 6 Pictured Figure 2 An example of the merging of facial key point metadata in the architecture 200 of FIG. Figure 6 Various changes may be made. For example, the number of facial landmarks 602, the number of facial landmarks 604, and the locations of facial landmarks 602 and 604 may be varied as needed or desired, depending on the implementation and the image(s) being processed.

[0096] Figures 7 to 11 1 shows example results that can be obtained using refined facial key point metadata generation for video conferencing or other applications according to the present disclosure. More specifically, Figures 7 to 11 Shows that you can use Figure 2 2. However, note that architecture 200 may be used in any other suitable manner.

[0097] like Figure 7As shown, a graphical user interface 700 may be presented to a user, such as a user of a sending device 202. Here, the graphical user interface 700 may include a control 702 that allows a user to take or load a selfie of the user, which may be used as a source image 206. The control 702 may allow the user to select a "frontalized" or "defrontalized" operation, both of which are described below. The graphical user interface 700 may include a user display area 704 that contains a live video of the user. During a video conference or other use, for example, the user display area 704 may present a captured driving video 208 of the user. For the convenience of the user, a marker 706 may optionally be presented in the user display area 704 to identify the area in which the user should attempt to place his or her face during a video conference or other use. This example of the user display area 704 may contain the actual face of the user, where the face of the user is partially obscured for privacy.

[0098] The graphical user interface 700 may include a composite user display area 708 that contains a composite image or composite video of the user. For example, the composite user display area 708 may present a single image representing how the user would appear (in a static manner) within a composite output video 254 generated by another device. The composite user display area 708 may also or alternatively present a composite output video 254 that would be generated by another device. This example of the composite user display area 708 may contain a composite face of the user, where the composite user's face is partially obscured here for privacy.

[0099] Note that while graphical user interface 700 includes both display areas 704 and 708, this is for ease of illustration and explanation only. In one or more embodiments of the present disclosure, sending device 202 may include graphical user interface 700 that presents only user display area 704 or presents user display area 704 and optionally presents composite user display area 708. Receiving device 202 may include graphical user interface that presents composite user display area 708 without presenting user display area 704, because receiving device 202 may not receive actual driving video 208.

[0100] Figure 8 An example of the artifact suppression capabilities of architecture 200 is shown in FIG. 2 , where a user has raised his hand near his face in driving video 208. Here, architecture 200 is able to sense the object and suppress metadata associated with the object. As a result, one or more other users may not see the user's hand because the metadata is corrected to remove metadata associated with the user's hand from one or more sets of refined metadata 242. As a result, Figure 8The corresponding portion of the composite output video 254 shown in FIG. 2 may lack the user's hand and contain little or no artifacts associated with the position of the user's hand.

[0101] exist Figure 7 and Figure 8 In the example shown, the synthesized face looks very much like the user's actual face, which can occur when the user uses an actual picture of the user as the source image 206. However, this is not required. Fig. 9 As shown, the synthetic user display area 708 can show an artificially generated face associated with the user. In some cases, the artificially generated face can represent a so-called "beautified selfie", which can represent an image of the user that has been modified in some way and used as the source image 206.

[0102] Regardless of the type of image used as source image 206, a "frontalization" operation or a "defrontalization" operation may be performed. A "defrontalization" operation may refer to an operation of architecture 200 in which the image of a user included in composite output video 254 generally follows the actual head movement of the real-world user. Fig.10 , where the user has actually turned his head to the left, and the image of the user as included in the composite output video 254 also turns to the left. The user here may make any suitable movements, and the head pose estimation function 230 may generally be used to track the user's head movements so as to allow the source image 206 of the user as included in the composite output video 254 to mimic the user's head movements. The source image 206 of the user included in the composite output video 254 will also be animated so that the user's facial expressions and lip movements mimic the user's actual facial expressions and lip movements.

[0103] In contrast, "frontalization" operation refers to operation of architecture 200 in which the image of a user included in composite output video 254 is generally facing forward, regardless of the user's actual head movement. Fig.11 254, where the user has actually turned his head to the left, but the image of the user contained in the synthesized output video 254 remains facing forward. This mode may be achieved by deactivating the head pose estimation function 230 or suppressing the rotation (and possibly translation) output from the head pose estimation function 230. This may cause the video rendering model 252 to generally keep the user's face forward during video conferencing or other use. However, the source image 206 of the user as contained in the synthesized output video 254 may still be animated so that the user's facial expressions and lip movements mimic the user's actual facial expressions and lip movements.

[0104] although Figures 7 to 11Example results that can be obtained using refined facial keypoint metadata generation for video conferencing or other applications are shown, but may be used for Figures 7 to 11 Various changes may be made. For example, graphical user interface 700 may have any other suitable form and include any desired content. In addition, the option of using the "frontalization" or "defrontalization" operation is not required, and architecture 200 may support one mode of operation but not another.

[0105] Fig.12 and 13 Example methods 1200 and 1300 for refined facial key point metadata generation and use according to the present disclosure are shown. For example, method 1200 can be used by sending device 202, and method 1300 can be used by receiving device 204. For ease of explanation, each method 1200 and 1300 is described as using Figure 1 However, each method 1200 and 1300 may be performed using any other suitable device and in any other suitable system. For example, method 1200 may be performed at least in part using Figure 1 The server 106 is used to execute.

[0106] like Fig.12 As shown, in step 1202, a source image of the user is obtained. This may include, for example, the processor 120 of the electronic device 101 capturing, receiving, or otherwise obtaining the source image 206 of the user. As described above, the source image 206 may be a current, recent, or old image of the user of the electronic device 101, or a beautified selfie or other artificial image of the user or associated with the user (which may or may not be generated by the electronic device 101). In step 1204, the source image may be used to identify the user's facial landmarks. This may include, for example, the processor 120 of the electronic device 101 executing the facial landmark estimation function 212 to identify the canonical facial landmarks 218 associated with the user's eyes, nose, mouth, cheeks, chin, forehead, and neck (or any combination thereof).

[0107] At step 1206, metadata associated with the source image may be identified based on the head pose and expression deformations of the user in the source image. This may include, for example, the processor 120 of the electronic device 101 executing the head pose estimation function 214 to identify rotation and translation parameters 220 associated with the user's head in the source image 206. This may also include the processor 120 of the electronic device 101 executing the expression deformation estimation function 216 to estimate one or more expression deformations 222 associated with the user's face in the source image 206.

[0108] At step 1208, the facial key points may be transformed using metadata associated with the source image. This may include, for example, the processor 120 of the electronic device 101 executing the transformation function 226 to transform the facial key points 218 using metadata 224 including rotation and translation parameters 220 and expression deformation 222 to produce source facial key points 228. At step 1210, the source image and the source facial key points may be sent. This may include, for example, the processor 120 of the electronic device 101 sending the source image 206 and the source facial key points 228 directly or indirectly (such as via the server 106) to another electronic device.

[0109] A driving video of the user may be obtained at step 1212. This may include, for example, the processor 120 of the electronic device 101 capturing, receiving, or otherwise obtaining a driving video 208 comprising a plurality of image frames 210. The user's face and head may typically move in the driving video 208, such as due to the user speaking, making expressions or gestures, or otherwise moving. Metadata associated with each image frame in the driving video may be identified at step 1214 based on the user's head pose and expression deformation in the image frame. This may include, for example, the processor 120 of the electronic device 101 executing a head pose estimation function 230 to identify rotation and translation parameters 234 associated with the user's head in each image frame 210. This may also include the processor 120 of the electronic device 101 executing an expression deformation estimation function 232 to estimate expression deformation 236 associated with the user's face in each image frame 210.

[0110] At step 1216, metadata associated with the one or more image frames may be refined to suppress one or more artifacts associated with one or more objects in the one or more image frames. This may include, for example, the processor 120 of the electronic device 101 providing a set of metadata 238 to a refinement model 240, which may identify noise metadata associated with the one or more image frames 210 and refine the noise metadata based on metadata associated with one or more earlier image frames 210. The refinement model 240 may generate a set of refined metadata 242, which may include refined rotation and translation parameters and refined expression deformations. At step 1218, the refined expression deformation-related metadata may be merged with the original expression deformation-related metadata for at least one of the image frames. This can include, for example, the processor 120 of the electronic device 101 executing a merge function 246 to replace the refined expression deformation-related metadata (such as generated by the refined model 240) with the original expression deformation-related metadata (such as generated by the expression deformation estimation function 232) for one or more facial landmarks (such as one or more facial landmarks on, near, or otherwise associated with or affected by the movement of the user's lips).

[0111] At step 1220, the facial key points may be transformed using the metadata associated with the image frame and the merged metadata. This may include, for example, the processor 120 of the electronic device 101 executing the transformation function 244 to transform the facial key points 218 using the set of refined metadata 242 and the merged metadata 248 to produce the streaming facial key points 250. The streaming facial key points may be sent at step 1222. This may include, for example, the processor 120 of the electronic device 101 sending the streaming facial key points 250 directly or indirectly (such as via the server 106) to another electronic device.

[0112] like Fig.13 As shown, a source image of a user, source facial features associated with the user, and streaming facial features associated with the user may be obtained at step 1302. This may include, for example, a processor 120 of one electronic device 101 receiving a source image 206, source facial key points 228, and streaming facial key points 250 from another electronic device 101, a server 106, or other device. In some cases, the source image 206 and source facial key points 228 may be initially received by the electronic device 101, and the streaming facial key points 250 may be received later in time after receiving the source image 206 and source facial key points 228.

[0113] At step 1304, the source image, source facial features, and streaming facial features may be provided to the video rendering model. This may include, for example, the processor 120 of the electronic device 101 providing the source image 206, the source facial key points 228, and the streaming facial key points 250 to the video rendering model 252. Again, in some cases, the source image 206 and the source facial key points 228 may be provided to the video rendering model 252 initially, and the streaming facial key points 250 may be provided to the video rendering model 252 over time.

[0114] At step 1306, an animation of the source image may be generated based on the source facial key points and the streaming facial key points. This may include, for example, the processor 120 of the electronic device 101 using the video rendering model 252 to determine how to adjust the source image 206 of the user based on the difference between the source facial key points 228 and the streaming facial key points 250. The determined adjustments to the source image 206 may be identified so that the source image 206 is changed so as to mimic at least some of the movements of the face or head of the user in the original drive video 208. The result may be a synthesized output video 254 that includes the user from the source image 206 animated by the video rendering model 252. Note that some or all of the movements of the face or head of the user in the original drive video 208 may be mimicked in the synthesized output video 254. For example, in a "de-frontalization" mode of operation, most or all of the movements of the face or head of the user in the original drive video 208 may be mimicked in the synthesized output video 254. In a “frontalization” mode of operation, movement of the user's face in the original drive video 208 may be mimicked in the composite output video 254 , but movement of the user's head may not be mimicked.

[0115] At step 1308, the composite output video may be stored, output, or used in some manner. This may include, for example, the processor 120 of the electronic device 101 presenting the composite output video 254 on an integrated or external display, such as on the display 160. If the composite output video 254 relates to a video conference or other use involving more than two users, the method 1200 may be performed for each participant, and the method 1300 may be repeated or performed in parallel to produce multiple composite output videos 254 for multiple participants presented to the user of the electronic device 101.

[0116] although Fig.12 and 13 Examples of methods 1200 and 1300 for refined facial key point metadata generation and use are shown, but may be used for Fig.12 and 13 Various changes are made. For example, while each figure is shown as a series of steps, the various steps in each figure may overlap, occur in parallel, occur in a different order, or occur any number of times.

[0117] Fig.14 1400 is shown, which is performed by an electronic device according to an embodiment of the present disclosure. Fig.14 According to an embodiment of the present disclosure, a method 1400 performed by an electronic device (eg, at least one processor of a device) may include operations 1410 to 1430. The method 1400 is not limited to Fig.14 The method shown in , and may also include Fig.14 Operations not shown in FIG.

[0118] In operation 1410, the electronic device may identify metadata of a source image including the appearance of a person or character. In an embodiment of the present disclosure, the metadata of the source image may include data of the source image (e.g., pixel data, image data) or at least one of data from the source image. In an embodiment of the present disclosure, the data from the source image may include at least one of: appearance features, canonical facial key points of a person or character, data of a head posture of a person or character (e.g., rotation data, translation data), data of a facial expression of a person or character (e.g., expression deformation data), facial key points of a person or character in the source image (e.g., source facial key points).

[0119] In operation 1420, the electronic device may obtain a target image including the appearance of the user. In an embodiment of the present disclosure, the user in the target image may be the same person as the person or character in the source image. In an embodiment of the present disclosure, the user in the target image may be a different person from the person or character in the source image.

[0120] In operation 1430, the electronic device may generate metadata of the target image with suppressed artifacts. In an embodiment of the present disclosure, the metadata of the target image may include at least one of data of a head posture of a user in the target image or data of a facial expression (i.e., expression deformation) of the user in the target image. In an embodiment of the present disclosure, the artifact may originate from an object that obscures at least a portion of the user's appearance in the target image or appears in the target image.

[0121] In an embodiment of the present disclosure, metadata for a source image and metadata for a target image may be a basis for generating a composite image whose appearance of a person or character mimics at least one of a user's facial expression or head posture. In an embodiment of the present invention, the metadata of the target image may include at least one of original metadata, refined metadata, merged metadata, or target facial key points.

[0122] In an embodiment of the present disclosure, an electronic device may send at least one of the metadata of a target image or the metadata of a source image to another electronic device (e.g., an end-user device) or a server. In an example, another electronic device (e.g., an end-user device) or a server may generate a composite image of a person or character whose appearance imitates at least one of a user's facial expression or head posture based on the metadata of the source image and the metadata of the target image. In an embodiment of the present disclosure, an electronic device may generate a composite image of a person or character whose appearance imitates at least one of a user's facial expression or head posture based on the metadata of the source image and the metadata of the target image. In an embodiment of the present disclosure, the electronic device may send the generated composite image to another electronic device or a server.

[0123] The computer-readable storage medium may be provided in the form of a non-transitory storage medium. In this regard, the term "non-transitory" simply means that the storage medium does not include a signal and is a tangible device, and the term does not distinguish between where data is semi-permanently stored in the storage medium and where data is temporarily stored in the storage medium. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0124] In addition, the program according to the embodiments disclosed in this specification may be included in a computer program product when provided. The computer program product may be traded between a seller and a buyer as a product. For example, the computer program product may be distributed in the form of a computer-readable storage medium (e.g., a compact disc ROM (CD-ROM)), or via an application store (e.g., Google Play Store TM ) online distribution (e.g., downloading or uploading) or directly between two user devices (e.g., smart phones). For online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be at least temporarily stored or temporarily created on a computer-readable storage medium (such as a memory of a manufacturer's server, an application store's server, or a relay server).

[0125] In an embodiment of the present disclosure, a method performed by an electronic device may include identifying metadata of a source image including the appearance of a person or character. In an embodiment of the present disclosure, a method may include obtaining a target image including the appearance of a user. In an embodiment of the present disclosure, a method may include generating metadata of a target image with suppressed artifacts. In an embodiment of the present disclosure, the metadata of the target image may include at least one of data of a head posture of the user in the target image or data of a facial expression of the user in the target image. In an embodiment of the present disclosure, the artifact may originate from an object that obscures at least a portion of the user's appearance in the target image or appears in the target image. In an embodiment of the present disclosure, metadata for the source image and metadata for the target image may be the basis for generating a composite image in which the appearance of a person or character mimics at least one of the facial expression or head posture of the user.

[0126] In an embodiment of the present disclosure, generating metadata of a target image may include extracting raw metadata of the target image from the target image. In an embodiment of the present disclosure, the raw metadata of the target image may include at least one of raw data of a user's head posture from the target image or raw data of a user's facial expression from an image frame. In an embodiment of the present disclosure, generating metadata of a target image may include using a trained machine learning model to refine at least a portion of the raw metadata of the target image to generate refined metadata of the target image. In an embodiment of the present disclosure, the trained machine learning model may be trained to suppress the influence of an object. In an embodiment of the present disclosure, the refined metadata may include at least one of refined data of a head posture or refined data of a facial expression.

[0127] In an embodiment of the present disclosure, the metadata of the source image may include one or more canonical facial key points of a person or character. In an embodiment of the present disclosure, generating metadata of a target image may include transforming one or more canonical facial key points based on the refined metadata to generate one or more target facial key points that reflect at least one of a user's head posture or facial expression. In an embodiment of the present invention, one or more target facial key points may be included in the metadata for the target image. In an embodiment of the present invention, a synthetic image may be generated based on the one or more target facial key points.

[0128] In an embodiment of the present disclosure, generating metadata for the target image may include merging the refined metadata with at least some of the original metadata to produce merged metadata for the target image. In an embodiment of the present disclosure, transforming one or more canonical facial key points may include transforming one or more canonical facial key points based on the merged metadata to produce one or more target facial key points.

[0129] In an embodiment of the present disclosure, the merging of the refined metadata with at least some of the original metadata may include replacing at least some of the refined metadata with at least some of the original metadata. In an embodiment of the present disclosure, at least some of the original metadata and at least some of the refined metadata may be associated with the user's lip expressions.

[0130] In an embodiment of the present disclosure, the data of the user's head posture may include at least one of the rotation data or translation data of the user's head posture in the target image. In an embodiment of the present disclosure, the trained machine learning model may include at least one of the following: a first gated recurrent unit (GRU)-based recurrent neural network trained to output refined rotation data of the head posture based on the original rotation data of the head posture, a second GRU-based recurrent neural network trained to output refined translation data of the head posture based on the original translation data of the head posture, or a third GRU-based recurrent neural network trained to output refined data of the user's facial expression based on the original data of the facial expression. In an embodiment of the present disclosure, the original rotation data may include at least one of the original pitch data, the original yaw data, or the original roll angle data. In an embodiment of the present disclosure, the refined rotation data may include at least one of the refined pitch data, the refined yaw data, or the refined roll angle data.

[0131] In an embodiment of the present disclosure, the target image may be one of multiple image frames of a driving video. In an embodiment of the present disclosure, the synthesized image may be one of multiple image frames of a synthesized video in which the appearance of a person or character imitates at least one of a facial expression or a head movement of a user in the driving video.

[0132] In an embodiment of the present disclosure, an apparatus may include at least one processor configured to identify metadata of a source image including the appearance of a person or character. In an embodiment of the present disclosure, an apparatus may include at least one processor configured to obtain a target image including the appearance of a user. In an embodiment of the present disclosure, a device may include at least one processor configured to generate metadata of a target image with suppressed artifacts. In an embodiment of the present disclosure, the metadata of the target image may include at least one of data of a head posture of the user in the target image or data of a facial expression of the user in the target image. In an embodiment of the present disclosure, the artifact may originate from an object that obscures at least a portion of the user's appearance in the target image or appears in the target image. In an embodiment of the present disclosure, the metadata for the source image and the metadata for the target image may be the basis for generating a composite image in which the appearance of a person or character may mimic at least one of the facial expression or head posture of the user.

[0133] In an embodiment of the present invention, the at least one processor may be configured to extract raw metadata of the target image from the target image. In an embodiment of the present disclosure, the raw metadata of the target image may include at least one of raw data of a head posture of a user of the target image or raw data of a facial expression of a user of the image frame.

[0134] In an embodiment of the present invention, the at least one processor may be configured to refine at least a portion of the original metadata of the target image using a trained machine learning model to generate refined metadata of the target image. In an embodiment of the present disclosure, the trained machine learning model may be trained to suppress the influence of an object. In an embodiment of the present disclosure, the refined metadata may include at least one of refined data of a head posture or refined data of a facial expression.

[0135] In an embodiment of the present disclosure, the metadata of the source image may include one or more canonical facial key points of a person or character. In an embodiment of the present disclosure, at least one processor may be configured to transform the one or more canonical facial key points based on the refined metadata to generate one or more target facial key points reflecting at least one of the user's head posture or facial expression. In an embodiment of the present invention, the one or more target facial key points may be included in the metadata for the target image. In an embodiment of the present invention, a synthetic image may be generated based on the one or more target facial key points.

[0136] In an embodiment of the present disclosure, at least one processor may be configured to merge the refined metadata with at least some of the original metadata to generate merged metadata for the target image. In an embodiment of the present invention, the at least one processor may be configured to transform the one or more canonical facial key points based on the merged metadata to generate the one or more target facial key points.

[0137] In an embodiment of the present disclosure, at least one processor may be configured to replace at least some of the refined metadata with at least some of the original metadata. In an embodiment of the present disclosure, at least some of the original metadata and at least some of the refined metadata may be associated with the user's lip expressions.

[0138] In an embodiment of the present disclosure, the data of the user's head posture may include at least one of the rotation data or the translation data of the head posture (for example, identifying the head posture of the user in the target image). In an embodiment of the present disclosure, the trained machine learning model may include at least one of the following: a first gated recurrent unit (GRU)-based recurrent neural network trained to output refined rotation data of the head posture based on the original rotation data of the head posture, a second GRU-based recurrent neural network trained to output refined translation data of the head posture based on the original translation data of the head posture, or a third GRU-based recurrent neural network trained to output refined data of the user's facial expression based on the original data of the facial expression. In an embodiment of the present disclosure, the original rotation data may include at least one of the original pitch data, the original yaw data, or the original roll angle data. In an embodiment of the present disclosure, the refined rotation data may include at least one of the refined pitch data, the refined yaw data, or the refined roll angle data.

[0139] In an embodiment of the present disclosure, the target image may be one of a plurality of image frames of a driving video. In an embodiment of the present disclosure, the synthesized image may be one of a plurality of image frames of a synthesized video, and the appearance of a person or character in the synthesized video may imitate at least one of a facial expression or a head movement of a user in the driving video.

[0140] In an embodiment of the present disclosure, a computer-readable medium may contain instructions that, when executed, cause at least one processor to identify metadata of a source image including the appearance of a person or character. In an embodiment of the present disclosure, the computer-readable medium may also contain instructions that, when executed, cause at least one processor to obtain a target image including the appearance of a user. In an embodiment of the present disclosure, a computer-readable medium may contain instructions that, when executed, cause at least one processor to generate metadata of a target image with suppressed artifacts. In an embodiment of the present disclosure, the metadata of the target image may include at least one of data of a head pose of the user in the target image or data of a facial expression of the user in the target image. In an embodiment of the present disclosure, the artifact may originate from an object that obscures at least a portion of the user's appearance in the target image or appears in the target image. In an embodiment of the present disclosure, the metadata for the source image and the metadata for the target image may be the basis for generating a synthetic image in which the appearance of a person or character mimics at least one of the facial expression or head pose of the user.

[0141] Although the present disclosure has been described with reference to various exemplary embodiments, various changes and modifications may be suggested to one skilled in the art. The present disclosure is intended to encompass such changes and modifications as fall within the scope of the appended claims.

Claims

1. A method performed by an electronic device (101), comprising: identifying metadata (1410) of a source image (206) including an appearance of a person or character; obtaining a target image (1420) including an appearance of a user; and generating metadata (1430) of the target image with artifacts suppressed, wherein the metadata of the target image includes at least one of data of a head pose of the user in the target image or data of a facial expression of the user in the target image, and the artifacts arise from an object that occludes at least a portion of the appearance of the user in the target image or appears in the target image, and The metadata of the source image (206) and the metadata of the target image are the basis for generating a composite image, wherein the appearance of the person or character in the composite image imitates at least one of the facial expression or head posture of the user.

2. The method according to claim 1, wherein: The generating metadata for the target image (1430) includes: extracting raw metadata of the target image from the target image, wherein the raw metadata includes at least one of raw data of the head pose of the user from the target image or raw data of the facial expression of the user from the image frame; and Refining at least a portion of the original metadata of the target image using a trained machine learning model to generate refined metadata of the target image, wherein the trained machine learning model is trained to suppress the influence of the object, and the refined metadata includes at least one of refined data of the head posture or refined data of the facial expression.

3. The method according to claim 1 or 2, wherein: The metadata of the source image includes one or more canonical facial landmarks of the person or the character, wherein generating metadata of the target image (1430) includes: transforming the one or more canonical facial key points based on the refined metadata to produce one or more target facial key points reflecting at least one of the head pose or the facial expression, and The one or more target facial key points are included in the metadata for the target image, and the composite image is generated based on the one or more target facial key points.

4. The method of claim 3, wherein generating metadata (1430) of the target image comprises: merging the refined metadata with at least some of the original metadata to produce merged metadata for the target image; as well as Wherein the transforming of the one or more canonical facial key points comprises transforming the one or more canonical facial key points based on the merged metadata to produce the one or more target facial key points.

5. The method according to claim 4, wherein: The merging of the refined metadata with the at least some original metadata comprises: replacing at least some of the refined metadata with the at least some original metadata; At least some of the original metadata and at least some of the refined metadata are associated with the user's lip expressions.

6. The method according to any one of claims 2 to 5, wherein the data of the head posture comprises at least one of rotation data or translation data of the head posture, and Wherein the trained machine learning model comprises at least one of the following: a first gated recurrent unit (GRU) based recurrent neural network (300) trained to output refined rotation data of the head pose based on raw rotation data of the head pose, wherein the raw rotation data comprises at least one of raw pitch data, raw yaw data, or raw roll angle data, and the refined rotation data comprises at least one of refined pitch data, refined yaw data, or refined roll angle data; a second GRU-based recurrent neural network (400) trained to output refined translation data of the head pose based on the raw translation data of the head pose; or A third GRU-based recurrent neural network (500) is trained to output the refined data of the facial expression based on the original data of the facial expression.

7. A method according to any one of claims 1 to 6, wherein the target image is one of multiple image frames of a driving video (208), and the composite image is one of multiple image frames of a composite video (254), and the appearance of the person or character in the composite video imitates at least one of the facial expression or head movement of the user in the driving video.

8. An apparatus comprising: At least one processor (120) is configured to: identifying metadata of a source image (206) including the appearance of a person or character; obtaining a target image including an appearance of a user; as well as generating metadata of the target image with suppressed artifacts, wherein the metadata of the target image comprises at least one of data of a head pose of the user in the target image or data of a facial expression of the user in the target image, and the artifacts originate from an object that occludes at least a portion of the appearance of the user in the target image or appears in the target image, and The metadata for the source image and the metadata for the target image are the basis for generating a composite image, wherein the appearance of the person or character in the composite image mimics at least one of a facial expression or a head posture of the user.

9. The device according to claim 8, wherein: The at least one processor is configured to: extracting raw metadata of the target image from the target image, wherein the raw metadata includes at least one of raw data of the head pose of the user from the target image or raw data of the facial expression of the user from the image frame; as well as Refining at least a portion of the original metadata of the target image using a trained machine learning model to generate refined metadata of the target image, wherein the trained machine learning model is trained to suppress the influence of the object, and the refined metadata includes at least one of refined data of the head posture or refined data of the facial expression.

10. The device according to any one of claims 8 or 9, wherein: The metadata of the source image includes one or more canonical facial landmarks of the person or the character, wherein the at least one processor is configured to: transforming the one or more canonical facial landmarks based on the refined metadata to produce one or more target facial landmarks reflecting at least one of the head pose or the facial expression, and The one or more target facial key points are included in the metadata for the target image, and the composite image is generated based on the one or more target facial key points.

11. The device according to any one of claims 8 to 10, wherein: The at least one processor is configured to: merging the refined metadata with at least some of the original metadata to produce merged metadata for the target image; as well as The one or more canonical facial key points are transformed based on the merged metadata to produce the one or more target facial key points.

12. The apparatus of any one of claims 8 to 10, wherein the at least one processor is configured to replace at least some of the refined metadata with at least some of the original metadata; and At least some of the original metadata and at least some of the refined metadata are associated with the user's lip expressions.

13. The device according to any one of claims 9 to 12, wherein the data of the head posture comprises at least one of rotation data or translation data of the head posture, and Wherein the trained machine learning model comprises at least one of the following: a first gated recurrent unit (GRU) based recurrent neural network (300) trained to output refined rotation data of the head pose based on raw rotation data of the head pose, wherein the raw rotation data comprises at least one of raw pitch data, raw yaw data, or raw roll angle data, and the refined rotation data comprises at least one of refined pitch data, refined yaw data, or refined roll angle data; a second GRU-based recurrent neural network (400) trained to output refined translation data of the head pose based on the raw translation data of the head pose; or A third GRU-based recurrent neural network (500) is trained to output the refined data of the facial expression based on the original data of the facial expression.

14. An apparatus according to any one of claims 8 to 13, wherein the target image is one of multiple image frames of a driving video (208), and the composite image is one of multiple image frames of a composite video (254), and the appearance of the person or character in the composite video imitates at least one of the facial expression or head movement of the user in the driving video.

15. A computer-readable medium comprising instructions which, when executed, cause at least one processor (120): identifying metadata of a source image including the appearance of a person or character; obtaining a target image including an appearance of a user; and generating metadata of the target image with suppressed artifacts, wherein the metadata of the target image comprises at least one of data of a head pose of the user in the target image or data of a facial expression of the user in the target image, and the artifact arises from an object that occludes at least a portion of the appearance of the user in the target image or appears in the target image, and The metadata for the source image and the metadata for the target image are the basis for generating a composite image, wherein the appearance of the person or character in the composite image mimics at least one of a facial expression or a head posture of the user.

Images