Small sample synthesis of talking heads

Abstract

Systems and methods are described for utilizing an image processing system having at least one processing device to perform operations including: receiving a plurality of input images of a user, generating a three-dimensional mesh proxy based on a first set of features extracted from the plurality of input images and a second set of features extracted from the plurality of input images. The method may also include: generating a neural texture based on the three-dimensional mesh proxy and the plurality of input images, generating a representation of the user including at least the neural texture, and sampling at least a portion of the neural texture from the three-dimensional mesh proxy. In response to providing the sampled at least a portion to a neural renderer, the method may include receiving from the neural renderer a composite image of the user that was not previously captured by the image processing system.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 198,287, filed on October 8, 2020, entitled “FEW-SHOTSYNTHESIS OF TALKING HEADS,” the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] The present description generally relates to methods, apparatus, and algorithms for synthesizing content for presentation. Background Art

[0004] Deep inverse rendering generally includes optimizing appearance parameters in a latent embedding space with a spatially varying appearance. Deep inverse rendering may include using a neural network to simulate the appearance of an object. Such a neural network may be used to learn a latent embedding space. The latent space may represent a space from which a low-dimensional object representation is obtained. Embedding such a space represents the way in which low-dimensional data is mapped to the original higher-dimensional image space. Summary of the invention

[0005] A system of one or more computers may be configured to perform a particular operation or action by installing software, firmware, hardware, or a combination thereof on the system, which, in operation, individually or collectively, causes the system to perform the action. One or more computer programs may be configured to perform a particular operation or action by including instructions that, when executed by a data processing device, cause the device to perform the action.

[0006] In one general aspect, systems and methods are described for utilizing an image processing system having at least one processing device to perform operations including receiving multiple input images of a user, generating a three-dimensional mesh proxy based on a first set of features extracted from the multiple input images and a second set of features extracted from the multiple input images. The three-dimensional mesh proxy may include a facial mesh proxy and a planar proxy. The method may further include generating a neural texture based on the three-dimensional mesh proxy and the multiple input images. Optionally, the method includes generating a representation of the user based at least in part on the three-dimensional mesh proxy and the neural texture. The method may further include sampling at least a portion of the neural texture from the three-dimensional mesh proxy. In response to providing at least a portion of the sampled to a neural renderer, the method may include receiving from the neural renderer a composite image of the user that was not previously captured (e.g., not seen) by the image processing system.

[0007] These and other aspects may include one or more of the following, alone or in combination.According to some aspects, the methods, systems, and computer-readable media claimed herein may include one or more (eg, all) of the following features (or any combination thereof).

[0008] In some embodiments, the first set of extracted features includes features representing a face of a user, and the second set of extracted features includes features representing hair and a portion of a torso of the user. In some embodiments, the neural texture is a learned neural texture generated using a plurality of input images, wherein the plurality of input images includes less than four captured image frames of the user.

[0009] In some embodiments, generating the neural texture includes: generating a latent representation for each input image; aggregating the representation generated for each input image into a vector; providing the vector to a texture generator network; and receiving a neural texture representing the user from the texture generator network. In some embodiments, generating the neural texture includes: projecting the extracted first set of features and second set of features into a geometrically perceptual feature space based on a depth image and a UV map associated with the input image, wherein the projection includes generating a set of images having a matching resolution of a plurality of input images. Generating the neural texture also includes: warping a set of images into a UV space to generate a texture for each input image; generating a score map for each point in the texture space for each texture, the score map indicating a visibility metric in the corresponding input image; applying a soft-max operation to each score map to generate an attention weight for each input image; and determining a weighted average based on the attention weights, aggregating the texture based on the weighted average.

[0010] In some embodiments, the generated representation of the user represents the appearance of each surface point represented in the neural texture, and wherein the method further comprises storing an appearance descriptor for each surface point to represent high frequency details about the appearance of the user. In some embodiments, the synthetic image is generated to depict the user through a corrected perspective for video conferencing, the corrected perspective simulating a view of the user looking at a camera device associated with a system providing video conferencing.

[0011] The implementation of the technology may include hardware, methods or processes or computer software on a computer-accessible medium. The details of one or more implementations are given in the following description and drawings. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram illustrating an exemplary 3D content system for displaying composite content on a display device according to an embodiment of the present disclosure.

[0013] Figure 2 is a block diagram of an exemplary system for compositing content for rendering on a display according to an embodiment of the present disclosure.

[0014] Figure 3A and Figure 3B An example of an enhanced mesh proxy generated by combining a facial mesh proxy and a plane proxy according to an embodiment of the present disclosure is shown.

[0015] Figure 4 is a block diagram of an exemplary architecture for generating composite content for rendering on a display according to an embodiment of the present disclosure.

[0016] Figure 5 is a block diagram of an exemplary architecture for generating neural textures based on an aggregated latent texture technique according to an embodiment of the present disclosure.

[0017] Figure 6 is a block diagram of an exemplary architecture for generating neural textures using inverse rendering techniques according to an embodiment of the present disclosure.

[0018] Figure 7 is a flowchart showing an example of a process of generating synthetic content using neural textures based on a 3D proxy geometric model according to an embodiment of the present disclosure.

[0019] Figure 8 Examples of computer devices and mobile computer devices that may be used with the techniques described herein are shown.

[0020] The same reference numbers in different drawings denote the same elements. DETAILED DESCRIPTION

[0021] In general, this document describes examples related to simulating novel (e.g., unseen) views of image content. For example, this document includes examples of simulating novel views of a talking head (e.g., a user's head and / or torso) using relatively few input images. A talking head may generally include views having one or more portions of a head, neck, and / or torso. As used herein, novel (e.g., unseen) views may include image content and / or video content that has been interpreted (e.g., synthesized, interpolated, simulated, etc.) based on one or more frames of image content and / or video content captured by a camera. Interpretation of image content and / or video content captured by a camera can be used in conjunction with the techniques described herein to create unseen versions and views (e.g., poses, expressions, angles, etc.) of the captured image content and / or video content.

[0022] When a high-quality three-dimensional (3D) view of the user is not available, it can be challenging to perform few-shot novel view synthesis to generate views of a user's head and / or body (e.g., torso). The systems and techniques described herein can provide a way to generate high-quality image content (e.g., images and / or videos) of novel torso views using a small number of images as input. In some embodiments, a small number of images can be used in combination with one or more 3D proxy planes and one or more neural textures to simulate 3D features, thereby enabling accurate rendering of unseen views of a 3D head and / or torso on a screen of a two-dimensional (2D) or 3D display. In some embodiments, the neural texture is based on an aggregated latent texture technique. In some embodiments, the neural texture is based on an inverse rendering technique. As used herein, the term "small sample" refers to the use of small sample machine learning techniques, in which machine learning is performed to understand new concepts from a small number of examples.

[0023] In some embodiments, the neural textures described herein may be used to generate unseen views in order to synthesize a video sequence of a user's speech and / or expression represented by the neural texture. For example, the neural texture may be used to generate a realistic head (and / or torso) image based on one or more images (e.g., (multiple) frames, videos, etc.). In some embodiments, the neural texture may be generated using a deferred neural rendering technique (e.g., a deep neural network), which may synthesize images by combining graphics with a learning component.

[0024] As used herein, a neural texture refers to one or more learned feature maps that are trained as part of the image capture process. For example, a neural texture can be generated by generating a feature map by mapping the features of the object to the 3D proxy geometry of the object as the object is captured. In some embodiments, a neural texture for an object can be generated based on a latent code associated with the view and capture pose of the object. In some embodiments, the neural texture is stored in one or more 2D maps on top of a 3D mesh. For example, such a neural texture may include a set of optimized feature maps learned during scene capture. In some embodiments, for example, a neural texture may encode a high-level description of the surface appearance of the captured object that can be interpreted by a deferred neural renderer.

[0025] In some embodiments, for example, the techniques described herein can be used to synthesize an image that is an accurate and realistic image for display on a screen of a 2D or 3D display used in a multi-channel 2D or 3D video conference. For example, the techniques and models described herein can be used to generate and display accurate and realistic views (e.g., image content, video content) of a torso and face. The views include unseen views that are traditionally difficult to depict in 3D. Conventional systems typically do not synthesize novel views of torso portions. For example, hair, neck, and shoulders may lack well-defined structure and therefore may not be represented by proxy geometry in conventional systems. The systems and methods described herein provide the advantage of simulating facial regions as well as regions outside of the facial region to ensure that the neural texture is robust against degradation of the proxy geometry. In particular, the systems and methods described herein enhance the facial mesh with a single plane as a coarse geometric proxy for the head region outside of the facial region.

[0026] In some embodiments, the techniques described herein can be used for entertainment purposes in movies, videos, short films, game content, or other formats that include a user's torso or face that can benefit from the small sample reconstruction techniques described herein. For example, the small sample reconstruction techniques described herein can be used to generate torso, face, and hair content for a moving character rendered in image and / or video content.

[0027] In some embodiments, the techniques described herein may be used by a virtual assistant device or other intelligent agent that can perform image processing to identify objects, recreate objects, and / or generate synthetic images based on these objects using the small sample reconstruction techniques described herein.

[0028] In some embodiments, the technology described herein uses a deep inverse rendering method to perform a small sample of novel views of a user. The technology can retain the visual identity of the user in the source image. For example, the technology can utilize a representation of the user including a 3D mesh proxy and (e.g., learned) neural texture. For example, the mesh proxy can be or include a polygonal (e.g., triangle) mesh. In particular, the technology described herein receives an input image including one or more frames of a user with any posture and expression. The received input image is used to calculate a potential representation of each frame of the user, where each frame is sampled using inverse rendering on a specific texture space. The potential representation can be further aggregated in the texture space using an attention mechanism that learns to fuse information from the input view to generate an improved and more complete texture compared to a traditional image generation system. Then, the technology described herein can generate novel and unseen views / image frames by sampling the texture using a facial mesh algorithm, which can be reprocessed using a neural renderer. The technology described herein provides the following advantages: while achieving accurate reconstruction of the user's torso, retaining user identification visual features.

[0029] Conventional systems can perform small sample view synthesis by encoding and aggregating the input into a single latent vector. This latent vector combines the shape, identity, and appearance of an object into a single vector. Because the appearance of each surface point is typically compressed into this latent vector, it may not be possible to represent and reconstruct special visual identity-specific information as well as high-frequency image details because the information is over-aggregated and cannot be separated and parsed out of a single latent vector. The systems and methods described herein factor the representation of an object (e.g., head, torso, user, etc.) into a 3D mesh proxy and a learned neural texture. In addition, the systems and methods use a neural texture to represent the appearance of the object at each surface point. Using a neural texture to store an appearance descriptor for each surface point provides a way to represent high-frequency image details and more identity-sensitive information.

[0030] In addition, the systems and methods described herein utilize a facial mesh algorithm to define a geometric representation of a user's face in conjunction with a coarse geometric proxy to represent the remainder of the user's head and / or torso structure. In some embodiments, the coarse geometric proxy includes at least one plane to represent hair, one or more shoulders, a neck, etc.

[0031] Figure 1 1 is a block diagram illustrating an exemplary 3D content system 100 for displaying content in a stereoscopic display device according to an embodiment of the present disclosure. For example, the 3D content system 100 may be used by multiple users to conduct 3D video conferencing communications (e.g., a telepresence session). In general, Figure 1The system can be used to capture video and / or images of a user during a 2D or 3D video conference and use the systems and techniques described herein to simulate the shape and appearance of the user's face and / or torso to render an accurate image depicting the user's face and / or torso in the video conference session.

[0032] System 100 can benefit from the use of the techniques described herein because such techniques can, for example, generate and display gestures, expressions, and portions of a user's image within a video conference that accurately represent the user in the video conference, but may be a computed latent representation of the actual captured image frames. The computed latent representation can be used with the techniques described herein to generate accurate textures and images of the user that can be displayed to another user, for example, in 2D and / or 3D via system 100.

[0033] like Figure 1 As shown, a first user 102 and a second user 104 are using a 3D content system 100. For example, the users 102 and 104 are participating in a 3D telepresence session using the 3D content system 100. In such an example, the 3D content system 100 may allow each user 102 and 104 to see a highly realistic and visually consistent representation of the other, thereby helping the users to interact in a manner similar to the physical presence of each other.

[0034] Each user 102, 104 may have a corresponding 3D system. Here, user 102 has 3D system 106, and user 104 has 3D system 108. 3D systems 106, 108 may provide functions related to 3D content, including but not limited to capturing images for 3D display, processing and presenting image information, and processing and presenting audio information. 3D system 106 and / or 3D system 108 may constitute a collection of sensing devices integrated into one unit. 3D system 106 and / or 3D system 108 may include reference Figures 2 to 6 as well as Figure 8 Part or all of the components described.

[0035] The 3D content system 100 may include one or more 2D or 3D displays. Here, a 3D display 110 is provided for the 3D system 106, and a 3D display 112 is provided for the 3D system 108. The 3D displays 110, 112 can use any of a variety of types of 3D display technology to provide an automatic stereoscopic view for each viewer (here, for example, user 102 or user 104). In some embodiments, the 3D displays 110, 112 can be independent units (e.g., self-supporting or hanging on a wall). In some embodiments, the 3D displays 110, 112 can include or have access to wearable technology (e.g., a controller, a head-mounted display, etc.). In some embodiments, the displays 110, 112 can be 2D displays.

[0036] In general, displays (such as displays 110, 112) can provide images that approximate the 3D optical properties of physical objects in the real world without using a head-mounted display (HMD) device. In general, the displays described herein include a flat panel display, a lenticular lens (e.g., a microlens array), and / or a parallax barrier to redirect images to multiple different viewing areas associated with the display.

[0037] In some embodiments, the displays 110, 112 may include high resolution and glasses-free lenticular 3D displays. For example, the displays 110, 112 may include a microlens array (not shown) including a plurality of lenses (e.g., microlenses) with glass spacers coupled (e.g., bonded) to the microlenses of the display. The microlenses may be designed such that, from a selected viewing position, a left eye of a user of the display may view a first set of pixels, while a right eye of the user may view a second set of pixels (e.g., wherein the second set of pixels is mutually exclusive of the first set of pixels).

[0038] In some exemplary displays, there may be a single location that provides a 3D view of image content (e.g., a user, an object, etc.) provided by such a display. A user can sit in the single location to experience appropriate parallax, minimal distortion, and realistic 3D images. If the user moves to a different physical location (or changes head position or eye gaze position), the image content (e.g., the user, an object worn by the user, and / or other objects) may begin to appear less realistic, 2D, and / or distorted. The systems and techniques described herein can reconfigure the image content projected from the display to ensure that the user can move around but still experience appropriate parallax, low distortion, and realistic 3D images in real time. Therefore, the systems and techniques described herein provide the following advantages: maintaining and providing 3D image content and objects for display to the user, regardless of any user movement that occurs while the user is viewing the 3D display.

[0039] like Figure 1 As shown, 3D content system 100 may be connected to one or more networks. Here, network 114 is connected to 3D system 106 and 3D system 108. Network 114 may be a publicly available network (e.g., the Internet) or a private network, just as two examples. Network 114 may be wired, wireless, or a combination of both. Network 114 may include or utilize one or more other devices or systems, including, but not limited to, one or more servers (not shown).

[0040] The 3D systems 106, 108 may include a number of components related to the capture, processing, transmission or reception of 3D information and / or the presentation of 3D content. The 3D systems 106, 108 may include one or more cameras for capturing image content of an image to be included in the 3D presentation. Here, the 3D system 106 includes cameras 116 and 118. For example, the camera 116 and / or the camera 118 may be substantially disposed within the housing of the 3D system 106, such that the objective lens or lens of each camera 116 and / or 118 captures the image content through one or more openings in the housing. In some embodiments, the cameras 116 and / or 118 may be separate from the housing, such as in the form of a stand-alone device (e.g., having a wired and / or wireless connection to the 3D system 106). The cameras 116 and 118 may be positioned and / or oriented so as to capture a sufficiently representative view of a user (e.g., user 102). Although cameras 116 and 118 generally do not obstruct the view of 3D display 110 of user 102, the placement of cameras 116 and 118 may be arbitrarily selected. For example, one of cameras 116, 118 may be placed somewhere above the face of user 102, and the other may be placed somewhere below the face. For example, one of cameras 116, 118 may be placed somewhere to the right of the face of user 102, and the other may be placed somewhere to the left of the face. For example, 3D system 108 may include cameras 120 and 122 in a similar manner. Additional cameras may also be present. For example, a third camera may be placed near or behind display 110.

[0041] In some embodiments, the 3D systems 106, 108 may include one or more depth sensors to capture depth data to be used in the 3D presentation. Such depth sensors may be considered part of the depth capture component in the 3D content system 100 for characterizing the scene captured by the 3D system 106 and / or 108 so that the scene is correctly represented on the 3D display. In addition, the system may track the position and orientation of the viewer's head so that the 3D presentation can be rendered with an appearance corresponding to the viewer's current perspective. Here, the 3D system 106 includes a depth sensor 124. In a similar manner, the 3D system 108 may include a depth sensor 126. Any of a variety of types of depth sensing or depth capture may be used to generate depth data.

[0042] In some embodiments, auxiliary stereo depth capture is performed. For example, a light spot can be used to illuminate the scene, and stereo matching can be performed between two corresponding cameras. This illumination can be accomplished using waves of a selected wavelength or wavelength range. For example, infrared light (IR) can be used. For example, in some embodiments, a depth sensor may not be used when a view is generated on a 2D device. The depth data may include or be based on any information related to the scene that reflects the distance between a depth sensor (e.g., depth sensor 124) and an object in the scene. For content in an image corresponding to an object in the scene, the depth data reflects the distance (or depth) to the object. For example, the spatial relationship between (multiple) cameras and the depth sensor may be known and can be used to associate images from (multiple) cameras with signals from the depth sensor to generate depth data for the image.

[0043] The images captured by the 3D content system 100 may be processed and then displayed as a 3D presentation. Figure 1 As shown in the example of , a 3D image of a face 104′, a torso, and hair 104″ is presented on a 3D display 110. In this way, the user 102 can perceive the 3D image 104′, torso, and hair 104″ as a 3D representation of the user 104, who can be away from the user 102. The 3D image 102′, torso, and hair 102″ is presented on a 3D display 112. In this way, the user 104 can perceive the 3D image 102′ as a 3D representation of the user 102.

[0044] 3D content system 100 can allow participants (e.g., users 102, 104) to engage in audio communications with each other and / or with other persons. In some embodiments, 3D system 106 includes a speaker and a microphone (not shown). For example, 3D system 108 can similarly include a speaker and a microphone. In this way, 3D content system 100 can allow users 102 and 104 to participate in a 3D telepresence session with each other and / or with other persons. In general, the systems and techniques described herein can operate with system 100 to generate image content and / or video content for display between users of system 100.

[0045] Generating image content to be displayed on the telepresence system 100 may include using any number of neural textures 134 and one or more proxy geometries, which may be mesh proxy geometries 136. As used herein, mesh proxy geometry represents two or more view-dependent textures mapped to a particular mesh geometry. For example, mesh proxy geometry 136 may represent a plurality of combined proxy geometries, such as a combination of a facial proxy geometry and a plane proxy geometry. In operation, the system 100 may access and / or otherwise utilize the system 200 to extract features from an input image, use these features and one or more proxy geometries to generate one or more neural textures 134, and synthesize one or more novel views based on the neural textures 134 and the mesh proxy geometry 136.

[0046] Figure 2 2 is a block diagram of an exemplary system 200 for synthesizing content for rendering according to an embodiment of the present disclosure. The system 200 may be used as or included in one or more embodiments described herein, and / or may be used to perform one or more examples of synthesizing, processing, simulating, or presenting image content described herein (multiple) operations. The entire system 200 and / or one or more of its individual components may be implemented according to one or more examples described herein.

[0047] System 200 may include one or more 3D systems 202. In the example shown, 3D systems 202A, 202B, through 202N are shown, where index N indicates an arbitrary number. 3D system 202 may provide for the capture of visual and audio information for 2D or 3D presentation, and forward the 2D or 3D information for processing. Such information may include an image of a scene, depth data about the scene, and audio from the scene. For example, 2D / 3D system 202 may be used as a system 106 and 2D / 3D display 110 ( Figure 1 ) or included therein.

[0048] System 200 may include multiple cameras, such as camera 204. Any type of light sensing technology may be used to capture images, such as the type of image sensors used in common digital cameras. Cameras 204 may be of the same type or of different types. For example, the camera positions may be placed anywhere on a 3D system (e.g., system 106).

[0049] System 202A includes a depth sensor 206. In some embodiments, depth sensor 206 operates by broadcasting IR signals onto a scene and detecting a response signal. For example, depth sensor 206 can generate and / or detect light beams 128A-B and / or 130A-B. In some embodiments, depth sensor 206 is an optional component, for example, in 2D video conferencing applications that do not utilize depth sensing. System 202A also includes at least one microphone 208 and speaker 210. In some embodiments, microphone 208 and speaker 210 can be part of system 106.

[0050] In addition, the system 202 includes a 3D display 212 that can present 3D images. In some embodiments, the 3D display 212 can be a stand-alone display, and in some other embodiments, it is a 3D display 212. In some embodiments, the 3D display 212 uses a parallax barrier technology to operate. For example, the parallax barrier can include parallel vertical stripes of a substantially opaque material (e.g., an opaque film) placed between the screen and the viewer. Due to the parallax between the viewer's respective eyes, different parts of the screen (e.g., different pixels) are viewed by the corresponding left and right eyes. In some embodiments, the 3D display 212 uses a double convex lens to operate. For example, alternating rows of lenses can be placed in front of the screen, and these rows direct the light from the screen to the left and right eyes of the viewer, respectively.

[0051] System 200 may include server 214, which may perform certain tasks of data processing, data simulation, data coordination, and / or data transmission. Server 214 and / or its components may include reference Figure 8 Part or all of the components described.

[0052] Server 214 includes texture generator 216, which may be responsible for generating 2D and / or 3D information in one or more ways. This may include receiving image content (e.g., from 3D system 202A), processing the image content, and / or forwarding the (processed) image content to another participant (e.g., to another of 3D systems 202).

[0053] According to the embodiments described in the present disclosure, the texture generator 216 can be trained by a generative latent optimization framework. Generally speaking, the texture generator 216 can use two or more 3D proxy geometries to parameterize the neural texture by using a generative model, which can generate various shapes and appearances of objects (e.g., users, subjects, expressions, characteristics, etc.).

[0054] In operation, texture generator 216 may represent an object by generating a feature map, such as feature map 238. Feature map 238 may represent the latent code of each object instance i as z i ∈R n The feature map 238 of the latent space may be an eight-dimensional (8D) map. For example, the feature map 238 may include random values ​​that are optimized using the system 200.

[0055] The feature maps 238 may be provided to one or more neural networks 244 to generate a plurality of neural textures 134. The neural textures 134 may represent portions of a mesh that define certain portions of the geometry and / or texture for a particular object represented in the feature maps 238.

[0056] The texture generator 216 may use a set of K proxies {P i,1 ,…,P i,K} (i.e., a triangular mesh with UV coordinates). For example, the architecture 400 can project a 2D image onto the surface of a 3D proxy model to generate a neural texture 134. The UV coordinates describe the axes of the 2D texture. The proxy is used to represent a version of the actual geometry of any or all sets of objects in the class.

[0057] In operation, the system 200 uses the principles of proxy geometry to encode geometry using a set of coarse proxy surfaces (eg, mesh proxy geometry 136 ) and a view-dependent neural texture 134 to encode shape, reflectivity, and view-dependent effects.

[0058] like Figure 2As shown, the texture generator 216 includes a texture processor / aggregator 218, a texture in-painter 220, a neural texture generator 222, an image warping engine 224, and a feature extractor 226. The texture processor / aggregator 218 can be configured to generate a neural texture using a small sample embedder network, such as an embedder network 245 of the neural network 244. In addition to the image latent representation (e.g., image latent vector 230), the embedder network can also be used to generate a per-image texture latent vector 228. The k-sample texture latents can then be aggregated by the texture processor aggregator 218. The aggregated texture latents can then be provided to a texture generator network (e.g., generator network 247) to obtain a neural texture 134 of a target subject (e.g., a user image). The texture latent vector(s) 228 and the image latent vector(s) 230 are shown in dashed form to indicate that each element can be optional in the system 200, for example, if the texture generator 216 generates the neural texture using a method that does not include an aggregated latent.

[0059] In some embodiments, the system 200 can construct a latent space for specific content (e.g., objects, features, image frames, etc.), and can feed the latent space for such content to the neural network 244, which can then, for example, generate a texture map 240 for each specific content type. In some embodiments, the system 200 can reduce the number of instances of the flat proxy from the training data to perform a few-shot reconstruction, while using the remaining proxies for the specific objects to train the class-level model of the neural network. For example, the remaining proxies can represent hairstyle images, which can be used to train the neural network 244 on a hair class. Such a class can be used to simulate new views of hair on a specific user image.

[0060] For example, in some embodiments, the texture processor / aggregator 218 may not use aggregated texture potentials, but may employ inverse rendering using UV maps 232 and mesh proxy geometry 136 to generate neural textures. The UV maps 232 may represent visible content included in a particular mesh proxy. The visible content may be used to generate one or more UV maps 232 based on texture coordinates. For example, the UV maps 232 may be provided to the feature extractor 226 to adjust the extraction of features in a particular set of images (extracted into feature maps 238). The UV maps 232 may enable the system 200 to allow the neural network 244 to obtain relevant data about a particular image and / or features represented in the image.

[0061] In some embodiments, the texture processor / aggregator 218 can generate a user representation (e.g., a latent representation stored in the texture latent vector 228 and / or the image latent vector 230) that takes into account specific user characteristics. For example, the aggregator 218 can obtain an input image (e.g., multiple frames of a person in any pose and expression) and can calculate a latent representation (e.g., a texture latent vector 228) for each input frame. Using inverse rendering on the texture space, the generated user representation (e.g., the texture latent vector 228 and / or the image latent vector 230) can be used to sample the image. The latent representation can be further aggregated in the texture space using an attention mechanism that learns to fuse information from the input views to create an improved and more complete texture 134. To generate a novel frame (e.g., a synthetic view 250), the neural texture 134 is sampled from the mesh proxy geometry 136 using the latent representation (e.g., the generated representation of the user), and the neural texture 134 is reprocessed using the neural renderer 248.

[0062] System 200 includes or has access to at least one neural network 244. As shown, neural network 244 includes at least an embedder network 245 and a generator network 247. Embedder network 245 includes one or more convolutional layers and downsampling layers. Generator network 247 includes one or more convolutional layers and upsampling layers.

[0063] The texture delineator 220 can generate content that may be missing from a particular texture based on a local neighborhood of pixels surrounding the particular missing content portion. In some embodiments, the texture delineator 220 can operate based on the fact that a large portion of the texture is symmetric about a vertical axis, and therefore, the missing portion of one half of the texture can be approximated based on the corresponding values ​​of the other half of the texture. The system 200 can concatenate the aggregated neural texture with a horizontally flipped version of the neural texture. The concatenated version of the neural texture can be provided to the texture delineator 220 to generate a final symmetric neural texture.

[0064] The neural texture generator 222 can generate a neural texture 134. The neural texture 134 represents a learned feature map 238 that is trained as part of the image capture process. For example, when an object is captured, the neural texture can be generated using the feature map 238 and one or more 3D proxy geometries 136 of the object. In operation, the system 200 can generate and store a neural texture 134 for a particular object (or scene) as a map on top of the one or more 3D proxy geometries 136 of the object. For example, the neural texture can be generated based on the latent code associated with each instance of the identified class and the view associated with the pose.

[0065] The visibility scores 236 may represent the visibility of a particular pixel or feature of a captured object. Each visibility score 236 may represent a single scalar value for each neural texture that indicates which portions of the image (e.g., pixels, features, etc.) are visible in a particular view of the input image. For example, if the far left side of a user's face is not visible in the input image of the user, the visibility score 236 representing the leftmost pixel of the user's face may be weighted low, while other areas that may be well viewed and / or captured in the input image may be weighted high.

[0066] The image warping engine 224 warps the image from the input image space (eg, feature space) to the texture space. For example, a particular input image is warped to the UV space to generate a neural texture (in the texture space).

[0067] Feature extractor 226 is a U-Net neural network that analyzes the image to generate features for each pixel. Feature extractor 220 can use such features to generate feature map 238 and texture map 240.

[0068] Sampler 246 may represent a 2D sampler that samples from a particular mesh proxy geometry for any novel view. The sampled texture is then provided to neural renderer 248.

[0069] The neural renderer 248 can generate intermediate representations of objects (e.g., users) and / or scenes, for example, to be rendered using the neural network 244. The neural texture 134 can be used to jointly learn features on a texture map (e.g., texture map 240) in conjunction with a U-Net (such as the neural network 244 operating with the neural renderer 248). For example, using an object-specific convolutional network, the neural renderer 248 can incorporate view-dependent effects by simulating the difference between the real appearance (e.g., ground truth) and the diffuse reprojection. Such effects can be difficult to predict based on scene knowledge, and therefore, a GAN-based loss function can be used to render realistic output.

[0070] In operation, the server 214 may obtain or receive an input image 242. The texture generator 216 may use one or more neural networks 244 to process the input image 242. The server 214 may use the texture generator 216, the sampler 246, and the neural renderer 248 to generate a synthetic view 250.

[0071] According to the following equation [1], the reconstruction loss (L CNT ), identity preservation loss (L ID ) and the opposing loss (L ADV ) to train the system 200 and the neural network 244 within the system 200 end-to-end:

[0072]

[0073] According to the following equation [2], based on the pre-trained VGG16 network and the pre-trained face algorithm grid network, the reference truth target image I is transformed using a mixture of L1 loss and perceptual loss. t and prediction The reconstruction error between is calculated as follows:

[0074] L CNT =λ L1 L1+λ VGG L VGG +λ FN L FN [2]

[0075] The system 200 can perform subject fine-tuning. For example, at test time, the small sample reconstruction can be applied to an unseen test subject. In some embodiments, the number of small samples need not be fixed and can vary between different test subjects. Given a set of small sample frames of a target object, the system 200 can perform simple fine-tuning of the neural network 244 or other networks within the system 200 to fit the target subject. Fine-tuning can be applied to any subset of the modules of the system 200. For example, fine-tuning can be applied to the texture generator 216 (denoted as G tex ), neural renderer 248 (denoted as R), and discriminator D (not shown). In addition, the system can fine-tune the network weights by directly optimizing the latent representation of a specific target subject. For example, the system 200 can treat the predicted neural texture as a free variable and can jointly optimize the neural network with the neural renderer 248 to better fit the target subject. Optimizing the neural texture can provide the advantage of being more interpretable than optimizing the latent vector representation because visualizing the neural texture can depict the high-frequency details added during the fine-tuning stage.

[0076] The above exemplary components are described as being implemented in server 214, which can be connected to server 214 through network 260 (which can be connected to server 214). Figure 1 114 in the example embodiment of the present invention) communicates with one or more 3D systems 202. In some embodiments, the texture generator 216 and / or its components may be implemented in part or all of the 3D system 202, alternatively or additionally. For example, the above simulation and / or processing may be performed by the system that originates the 3D information before forwarding the 3D information to one or more receiving systems. As another example, the initiating system may forward the image, simulation data, depth data, and / or corresponding information to one or more receiving systems, which may perform the above processing. Combinations of these methods may be used.

[0077] System 200 is an example of a system that includes a camera (e.g., camera 204), a depth sensor (e.g., depth sensor 206), and a 3D content generator (e.g., texture generator 216) that causes a processor to execute instructions stored in a memory. Such instructions may cause the processor to use depth data included in the 3D information (e.g., by a depth processing component) to identify image content in an image of a scene included in the 3D information. The processor may generate modified 3D information by sampling neural texture 134, for example, which may be provided to texture generator 216 to correctly depict synthetic view 250.

[0078] As described herein, based at least in part on the augmented face mesh, synthetic view 250 represents a 3D stereoscopic image of a particular object (e.g., user image 412, 512, 610) with appropriate disparity and viewing configuration for both eyes associated with a user accessing a display (e.g., display 212). For example, each time the user uses system 200 while moving the position of the head while viewing the display, at least a portion of synthetic view 250 can be determined based on output from neural network 244. In some implementations, synthetic view 250 represents the user's face and other features of the user surrounding the user's face within the view that captures the user's face.

[0079] In some embodiments, the processor (not shown) of systems 202 and 214 may include (or communicate with) a graphics processing unit (GPU). In operation, the processor may include or may access memory, storage, and other processors (e.g., CPU). To help generate graphics and images, the processor may communicate with the GPU to display images on a display device (e.g., display device 212). The CPU and GPU may be connected via a high-speed bus, such as PCI, AGP, or PCI-Express. The GPU may be connected to the display via another high-speed interface (e.g., HDMI, DVI, or Display Port). Typically, the GPU may render image content in pixel form. The display device 212 may receive image content from the GPU and may display the image content on a display screen.

[0080] Figure 3A is a block diagram of an example of an augmented mesh proxy 300 generated by combining an augmented face mesh proxy and a plane proxy as a coarse geometric proxy according to an embodiment of the present disclosure. The augmented mesh proxy 300 may be used to synthesize image and / or video content of a user, including but not limited to facial features, hair features, and / or torso features associated with the user.

[0081] Augmented mesh proxy 300 may be generated by combining two or more proxy geometry representations. For example, augmented mesh proxy 300 may be generated by combining a representation of a user's face determined using 3D augmented facial mesh proxy 302 with a planar proxy 304, which in this example represents a portion of the user's hair and / or the user's torso (e.g., the remainder of the user's head / torso portion not included in facial mesh proxy 302). For example, a facial mesh algorithm may be combined with a single planar geometry proxy to generate an accurate synthesis of a user's head and torso image. For example, as Figure 1 As shown, the facial mesh proxy 302 and the plane proxy 304 may be represented as mesh proxy geometry 136 .

[0082] The facial mesh proxy 302 may represent texture coordinates corresponding to visible content in the input image. For example, the visible content may include a facial portion of the proxy 302, but not necessarily a background surrounding the facial portion of the proxy 302. The visible content may be used to generate a UV map (e.g., UV map 232) from the texture coordinates. For example, the UV map 232 may be provided to the feature extractor 226 to adjust the extraction of features in a particular set of images. The UV map 232 may enable the system 200 to allow the neural network 244 to obtain relevant data about a particular image and / or features represented in the image.

[0083] A convolutional neural network can be used to generate a facial mesh. Given an image, the convolutional neural network outputs a series of coefficients that represent the 3D positions of a set of specified vertices, with a predefined topology representing the face. The 3D positions are such that they are aligned with the spatial coordinates of the input image, while the depth dimension can be arbitrary or in metric units.

[0084] Planar proxy 304 is depicted as a plane placed to include the user's face (e.g., in a view perpendicular to the plane) and surrounding areas (e.g., hair, torso, etc.). Planar proxy 304 represents a planar billboard that simulates the user's head. The systems and techniques described herein can utilize other objects and planar proxy shapes that represent these contents to generate and render 3D content. For example, other proxies may include, but are not limited to, boxes, cylinders, spheres, triangles, etc.

[0085] A plane proxy can represent a texture-mapped object (or part of an object) that can be used as a substitute for complex geometry. Because manipulating and rendering geometric proxies is computationally less than manipulating and rendering the corresponding detailed geometry, a plane proxy representation can provide a simpler shape for reconstructing a view. A plane proxy representation can be used to generate such a view. When trying to manipulate, reconstruct and / or render objects with a highly complex appearance (such as glasses, cars, clouds, trees, and grass, just as some examples), using a plane proxy can provide the advantage of low computational cost. Similarly, due to the availability of powerful graphics processing units, real-time game engines can use these proxies (e.g., geometric representations) with multiple levels of detail that can be exchanged in and out with distance by using 3D proxy geometry generation mapping to replace geometry with lower levels of detail.

[0086] In operation, the system 200 can generate a plane proxy 304, for example, by computing a bounding box (e.g., a roughly visible shell) of the head portion using the extracted alpha mask. In general, for any number of pixels in the head portion, the alpha mask represents how a particular pixel color merges with other pixels when overlapped. The system 200 can then specify a region of interest in the image of the user's head. For example, the region of interest can be specified using head coordinates. The system 200 can then extract a plane that probabilistically matches the surface when viewed from a corresponding orthogonal projection.

[0087] In general, the system 200 can generate planar proxies for any number of images that can be used as training data for input to a neural network (e.g., neural network 244). For example, the neural network can determine how to properly display a particular object (e.g., a head part, a face part, etc.) captured by a camera. In some embodiments, for example, the system 200 can generate a view of a particular head part by assembling an image dataset that includes the head part and simulating the head part from a gesture-based perspective using a detected gesture.

[0088] The augmented mesh proxy 300 can be used to ensure accurate representation and reproducibility of the face and other head features associated with the user's head. For example, in order to include areas other than the face area in the synthesized image, the augmented mesh proxy 300 can be used to generate a neural texture that is robust against degradation of the synthesized image. Such a neural texture can be generated by combining data representing 3D transformations and data representing image perspective and learned rendering techniques to synthesize new image content.

[0089] Figure 3BAn exemplary input for generating an enhanced mesh agent according to an embodiment of the present disclosure is shown. For example, an input image 320 may be obtained. For example, the input image 320 may represent a target subject (e.g., a user). The input image may be used to generate multiple texture maps. Texture maps may represent 2D and 3D images of a user. Texture maps are represented using 2D images to generate a 3D model in UV space, where "U" and "V" represent 2D texture axes. Texture maps may be used by the system 200, for example, to map a function onto a 3D surface. The function domain may be 1D, 2D, or 3D, and may be represented as a mathematical function (e.g., an array). Texture space may be defined by a UV space that may be mapped to an object space (e.g., an x, y, and z coordinate space). The system 200 may also use texture mapping to predict new texture coordinates.

[0090] For example, the system 200 may generate a face mesh texture map 322, which may be used to generate the face mesh proxy 302. Similarly, the system 200 may generate a head proxy U texture map 324, which may be used to generate the plane proxy 304. The texture map generated by the system 200 may be used to generate a neural texture. For example, the system 200 may use the face mesh proxy 302 and the face mesh texture map 322 to generate a neural texture 326 representing the face of the user 320. Similarly, the system 200 may use the plane proxy 304 and the head proxy texture map 324 to generate a neural texture 328 representing the rest of the head of the user 320 (e.g., hair, torso, etc.). Thus, the system 200 may generate a learned neural texture for the user by combining the face mesh texture 326 with the head proxy texture 328. In general, the appearance of each surface point is represented in the learned neural texture. Storing an appearance descriptor for each surface point provides the advantage of representing high-frequency details and accurate identity-sensitive information.

[0091] Figure 4 4 is a block diagram of an exemplary architecture 400 for generating synthetic content for rendering according to an embodiment of the present disclosure. The architecture 400 includes a plurality of input images 402 (e.g., input minisamples), which may be as few as three input images. The architecture 400 also includes a neural texture generator 404 to generate a neural texture 406. The architecture 400 also includes a 2D sampling module 408, a neural renderer 410, which may be used in conjunction with the neural texture generator to generate a synthetic view 412 of a user shown in the input image 402. For example, the neural renderer 410 is a renderer that includes and / or uses a neural network.

[0092] In operation, given a small sample image (e.g., input image 402) of a target object (e.g., a user), the system 200 can perform novel view synthesis of the user in the input image 402. For example, the system 200 can process the input image 402 using a neural texture generator 404, and aggregate the processed image to construct an enhanced mesh proxy 300 and a learned neural texture 406 of the target object (e.g., the user of the input image 402). The system 200 can then sample the learned neural texture 406 from the proxy geometry and input the sampled points (e.g., performed by a sampling module 408) to a neural renderer network (e.g., neural renderer 410), which can then synthesize a target view of the user in the input image 402.

[0093] The system 200 may implement the architecture 400 using a first architecture 500 that includes an encoder-decoder architecture to generate a neural texture 406 of a target object in the input image 402. Alternatively, the system 200 may implement the architecture 400 using a second architecture 600 that uses an inverse rendering technique to generate a neural texture 406 of a target object in the input image 402.

[0094] Figure 5 5 is a block diagram of an exemplary architecture 500 for generating neural textures using the aggregated latent texture technique according to an embodiment of the present disclosure. For example, neural texture generator 222 can be used to learn neural textures (e.g., neural texture 510) from neural network 244. Texture generator 216 of system 200 can be adjusted, for example, using Figure 5 The architecture 400 may be implemented by generating sampled neural textures from components of the texture generator 216. The texture generator 216 may represent an image synthesis network that may be conditioned on a sampled neural texture (e.g., neural texture 510) rather than on a conventional system that samples sparse 2D landmarks within an image.

[0095] To generate a sampled neural texture, the system 200 may obtain an input image 502 (described herein as an image {I1, I2 . . . I k}). The input image 502 may include multiple views (e.g., poses) of a user captured by a camera device. The input views may be provided to an embedder network, such as the embedder network 245. The embedder network 245 may represent a mapping of the input image 502 (e.g., face, torso, or any combination of user parts) to an embedding vector 506 (described herein as {z1, z2 . . . z k}), the embedding vector 506 generally stores information that is independent of posture. In some embodiments, the system 200 can provide the spatial input 504 to the embedder network 245. For example, the spatial input 504 can be represented as Figure 3AThe proxy geometry of the 3D augmented facial mesh proxy 302 is combined with the plane proxy 304 to create the spatial input 504 .

[0096] For example, the embedding vectors 506 may be aggregated using the aggregator 218. The result of aggregating the vectors 506 may be an aggregated vector 508 (shown here as z). The aggregated vector 508 may be provided to the texture generator network 247. For example, the texture generator network 247 represents a network that may map the aggregated embedding vectors to generate a neural texture 510. The output neural texture 510 may be used with the architecture 400 (using components of the system 200) to generate a novel synthetic view 512.

[0097] In operation, a per-image texture latent vector may be generated by configuring the (small sample) embedder network 245(E) To generate neural texture 510. For i∈{1…k}, system 200 can potentially represent the image as a k-sample texture The k sample texture potentials (shown as vector 506) are then aggregated using an averaging equation as shown in the following equation [3]:

[0098]

[0099] The aggregate texture potential (shown as vector 508) is then provided to the (texture) generator network 247 (eg, G tex ), the generator network 247 outputs the target object T = G tex z tex A neural texture 510 (e.g., T).

[0100] Figure 6 6 is a block diagram of an exemplary architecture 600 for generating a neural texture (e.g., neural texture 602) using an inverse rendering technique according to an embodiment of the present disclosure. The inverse rendering technique may include any number of sub-processing modules. In the following example, at least three sub-processing modules are used. The first sub-processing module may include a feature extractor network 226. The second sub-processing module may include an image warping engine 224. The third sub-processing module may include a texture processor / aggregator 218.

[0101] Given a small sample image of a subject (e.g., a user), the system 200 can perform an inverse rendering technique to construct a corresponding neural texture for the user. The inverse rendering techniques described herein may include estimating physical properties of the scene (e.g., reflections, geometry, lighting, etc.) relative to the image of the user (e.g., the user's head, neck, hair, torso, etc.). In an exemplary operation of the architecture 600, the feature extractor network F(.; φ) 226 may receive an input image (e.g., a red-green-blue (RGB) frame) ), which is shown here as a corresponding depth image and UV mapping The enhanced image 603 (i.e., I1, I2 ... I k ).

[0102] The feature extractor network 226 may project each of the k sample inputs to a region corresponding to the input image x 1…k A geometry-aware feature space with the same resolution, where:

[0103]

[0104] Next, the projected image is warped into UV space to generate a neural texture y for each input image. i ∈ d (e.g., texture 134). The texture processor network P(.;ω) for each neural texture y i (e.g., texture 134) is post-processed and also outputs an additional score map S i , where the score at each point in texture space indicates its visibility in the corresponding input image (e.g., visibility score 236), as shown in the following equation [5]:

[0105]

[0106] For example, feature extractor 226 may generate one or more feature maps 238, represented as a plurality of feature maps 238 (e.g., Figure 6 ). The feature map 238 represents the features of the input image in the feature space. For example, for each input image 603, the feature map 238 can be generated using the extracted image features.

[0107] For example, the image warping engine 224 may be used to warp each feature map 238 to generate a plurality of texture maps 240. For example, if three input images are provided to the system 200, the feature extractor 226 may extract three feature maps, one for each input image 603. The image warping engine 224 may warp each of the three feature maps 238 into a texture map 240, thereby generating three texture maps, such as Figure 6 shown.

[0108] Next, the neural texture generator 222 using the texture processor / aggregator 218 may apply a soft-max to the k visibility scores 236 (eg, maps) to obtain a per-point attention weight map 608 {A1 ... A k Based on the predicted attention weights, the k neural textures are aggregated into a weighted average to represent the aggregated k neural textures by the attention weights and visibility scores, which are aggregated in the texture space as shown in the following equation [6]:

[0109]

[0110] Aggregate Texture The missing portion of the texture may include missing portions that are not visible in any input view. The missing portion of the texture may be drawn based on its determined local neighborhood. Alternatively, the missing portion of the texture may be drawn by assuming that most of the texture is symmetric around a vertical axis and assuming that half of the missing portion can be approximated from the corresponding values ​​of the other half. Therefore, the texture descriptor 220 may concatenate the aggregated neural texture with a horizontally flipped version (e.g., 90 degree rotated) of the aggregated neural texture. The concatenated version of the texture may be provided to a texture descriptor network G(.; ψ) (e.g., the texture descriptor 220), which may output a final neural texture 602. The following equation [7] represents the concatenated version of the texture:

[0111]

[0112] Here, | represents channel-wise concatenation, and hor_flip represents horizontal flipping around the vertical axis (e.g., 90-degree rotation).

[0113] Neural texture 602 represents a learned feature map trained on a neural network. In this example, the neural texture includes one or more maps overlaid on a 2D or 3D mesh proxy. Portion 604 of neural texture 602 represents a texture generated using a facial mesh proxy (e.g., 302), while portion 606 represents a texture generated using a planar proxy (e.g., 304) to capture hair and / or torso portions. The three images in portion 604 may represent a nine-channel texture of three RGB color images with a first set of feature points of input image 603. The three images in portion 606 may represent a nine-channel texture of three RGB color images with a second and different set of feature points of input image 603. In addition, visibility score 236 (e.g., a map) may be used as another information channel for the texture represented by neural texture 602.

[0114] After completing the generation of the predicted neural texture T (e.g., neural texture 602), the system 200 can use the sampler 246 to extract the predicted neural texture T from the mesh proxy geometry (such as Figure 3A The neural texture 602 is sampled from the enhanced mesh proxy geometry 300 shown in FIG. 6 . Here, the enhanced mesh proxy geometry 300 includes the use of the face mesh proxy geometry 302 and the plane proxy geometry 304. The enhanced mesh proxy geometry 300 can be sampled for any novel view. The resulting sampled texture can be provided to the neural renderer 248 (e.g., the neural renderer network R(.; θ)) that can synthesize the view 610, as shown in the following equation [8]:

[0115] in

[0116] The neural renderer network R248 can use the low-dimensional per-point descriptors of the neural texture 602 to recover the user's object geometry and appearance.

[0117] Figure 7 8 is a flowchart illustrating an example of a process 700 for generating synthetic content using neural textures based on a 3D proxy geometry model according to an embodiment of the present disclosure. In short, for example, the process 800 can provide an example of using a 3D proxy geometry with a mesh algorithm to generate a small sample of novel views of 2D and 3D images that the user has not seen.

[0118] Process 700 may utilize an image processing system having at least one processing device and a memory storing instructions that, when executed, cause the processing device to perform a plurality of operations and computer-implemented steps as described in the claims. In general, systems 100, 200, and / or architectures 400, 500, and / or 600 may be used for the description and execution of process 700. In some embodiments, each system 100, 200 and architecture 400, 500, and / or 600 may represent a single system.

[0119] In general, process 700 utilizes the systems and algorithms described herein to obtain or retrieve a small sample image of a target subject and perform novel view synthesis. For example, process 700 may include processing and aggregating small sample inputs to construct a 3D mesh proxy with a learned neural texture of the target subject. In order to synthesize a novel view, the learned texture can be sampled from the proxy geometry and passed to a neural renderer network that synthesizes the target view. In some embodiments, an encoder-decoder architecture can be used to generate a neural texture for the target subject. In some embodiments, an inverse rendering technique can be used to generate the neural texture.

[0120] At block 702, process 700 includes receiving a plurality of input images representing a user (e.g., input sample 402). For example, server 216 may receive, obtain, and / or capture images of user 104 in real time or from a previous video conferencing activity and / or other image capture activity. The input images may include the user's face, features, hair, neck or other torso portion, etc. Generally, such input images may be a small sample of less than four images used in process 700. In some embodiments, three input images are used to perform the steps of process 700. In some embodiments, a single image may be used to perform the steps of process 700.

[0121] At block 704, process 700 includes generating a 3D mesh proxy (e.g., augmented mesh proxy 300) based on a first set of features extracted from the plurality of input images 242 (e.g., stored in feature map 238). For example, texture generator 216 may generate feature map 238 using a first set of features representing a face of a user. The first set of features may include a portion of a face mesh, such as face mesh proxy 302. 3D mesh proxy 300 may be further generated based on a second set of features extracted from the plurality of input images 242 (e.g., stored in feature map 238). The second set of features may include features representing a portion of the user's hair and / or torso. The second set of features may be retrieved from a plane proxy, such as plane proxy 304.

[0122] At block 706, process 700 includes generating a neural texture (e.g., neural texture 406) based on the 3D mesh proxy and the plurality of input images. For example, a neural texture generator (such as generator 404 or neural texture generator 222) may generate a 3D mesh proxy 406 of the user using a combination of a face mesh proxy 302 (shown in mapping 322) and a plane proxy 304 (shown in mapping 324). 3D mesh proxy 406 may include portion 326 representing face mesh texture 326 and portion (i.e., plane proxy) representing head proxy texture 328.

[0123] At box 708, process 700 includes generating a representation of the user. The representation of the user can be based at least in part on a 3D mesh proxy and a neural texture. For example, the texture processor aggregator 218 can generate a mesh proxy geometry, and the neural texture generator 222 can generate a neural texture. Thus, the system 200 can generate a first set of features representing the user's face and a second set of extracted features representing the user's hair and torso, both of which can be based on the mesh proxy and the learned neural texture. In general, the representation of the user represents the appearance of each surface point represented in the neural texture 134. This appearance is visible in some places of the neural texture and not visible in other places of the neural texture. Therefore, the visibility score 236 can be used when sampling or otherwise accessing the neural texture. In some embodiments, the system 200 can store an appearance descriptor for each surface point to represent high-frequency details about the user's appearance, or view-dependent effects of the user's appearance, such as specular highlights on the skin.

[0124] At block 710, process 700 includes sampling at least a portion of a neural texture from the 3D mesh proxy 300. For example, sampler 246 may perform 2D sampling of the neural texture. That is, sampler 246 may sample neural texture 406 from the proxy geometry (represented as combined 3D mesh proxy 300). In some embodiments, a predicted neural texture is sampled from the mesh proxy for any novel view. In some embodiments, sampler 246 represents a 2D sampler that may sample from a neural texture using UV mapping 232. In some embodiments, neural texture 406 is a learned neural texture generated and sampled from a plurality of input images 402, which may include less than four captured user image frames.

[0125] At block 712, process 700 includes providing at least one sampled portion to a neural renderer, such as neural renderer 248. In response, system 200 may receive from neural renderer 248 a synthetic image of the user that was not previously seen by the image processing system (e.g., synthetic view 250, synthetic view 412, etc.). For example, the synthetic image may include a view that the image processing system has not yet recorded (e.g., viewing the user at an angle, and / or the user holding the head and / or looking in a certain direction). For example, synthetic view 250 may be triggered for rendering on device 110 or 112.

[0126] In some embodiments, a composite image is generated to depict the user with a corrected perspective (i.e., relative to the direction of eye gaze of the remote user) for the video conference. Thus, the corrected perspective can make it appear as if two users looking at each other in the video conference are maintaining direct eye contact with each other. For example, the composite view of the user can be a similar pose of the user in the input image, but with a corrected perspective, thereby simulating a view of the user looking at a camera device associated with the system providing the video conference. This correction can provide a sense of presence for the two users of the video conferencing system, as the system 200 can provide the appearance of eye contact between the two users accessing the video conferencing system.

[0127] In some embodiments, generating a neural texture includes aggregating potential users, such as Figure 5 As shown and described in . For example, the texture processor / aggregator 218 can generate a latent representation for each input image 242, 502 (e.g., image frame). The latent representation generated for each input image (e.g., k-sample texture latent) can be aggregated into a vector 508 using an averaging technique. For example, the vector 508 can be provided to the texture generator network 247. A neural texture 510 representing the user can be received from the texture generator network 247. The neural texture 510 can be used to generate a synthetic view 512.

[0128] In some embodiments, generating a neural texture comprises using an inverse rendering algorithm, such as Figure 6 As shown and described in . For example, an input image can be provided to a feature extractor network 226, which can extract at least a first set and a second set of features to enhance the image 603 through a corresponding depth image and UV map 232. The system 200 can then project the extracted first set of features and the second set of features into a geometry-aware feature space based on the depth image (captured by the depth sensor 206) and the UV map 232 associated with the input image 603. The projection may include generating a set of images having matching resolutions of the multiple input images 603. For example, the system 200 can project each k sample input to a geometry-aware feature extractor having the same resolution as the input image 603. For example, the system 200 can warp the set of images to UV space to generate a texture for each input image 603, as shown by the K neural textures 134 ( Figure 6 ).

[0129] Next, the texture processor and aggregator 218 can generate a score map (e.g., visibility score 236) for each point in the texture space for each texture, which indicates a visibility metric in the corresponding input image. In some embodiments, the visibility score 236 (e.g., score map) is generated by processing the neural texture of each image. The visibility score 236 at each point in the texture space indicates its visibility in the corresponding input image.

[0130] The system 200 can apply a soft-max operation to each score map (e.g., visibility score 236) to generate attention weights for each input image. A weighted average can then be determined based on the attention weights, and textures can be aggregated based on the weighted average. In some embodiments, additional steps can be performed by the texture delineator 220 to generate missing parts that are not visible in any input image 603. Figure 2 Details of the texture painter 220 execution are described.

[0131] Figure 8 An example of a computer device 800 and a mobile computer device 850 that can be used by the described technology is shown. The computing device 800 may include a processor 802, a memory 804, a storage device 806, a high-speed interface 808 connected to the memory 804 and a plurality of high-speed expansion ports 810, and a low-speed interface 812 connected to a low-speed bus 814 and the storage device 806. The components 802, 804, 806, 808, 810, and 812 are interconnected using various buses and can be mounted on a common motherboard or installed by other means as appropriate. The processor 802 can process instructions for execution in the computing device 800, including instructions stored in the memory 804 or on the storage device 806, to display graphical information for a GUI on an external input / output device (such as a display 816 coupled to the high-speed interface 808). In some embodiments, multiple processors and / or multiple buses can be used as appropriate along with multiple memories and multiple types of memories. Furthermore, multiple computing devices 800 may be connected, with each device providing a portion of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0132] The memory 804 stores information in the computing device 800. In one embodiment, the memory 804 is a volatile memory unit or multiple volatile memory units. In another embodiment, the memory 804 is a non-volatile memory unit or multiple non-volatile memory units. The memory 804 can also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0133] The storage device 806 can provide mass storage for the computing device 800. In one embodiment, the storage device 806 can be a computer-readable medium or can include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state storage device, or an array of devices, including devices in a storage area network or other configuration. A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that perform one or more methods (such as the methods described herein) when executed. The information carrier can be a computer-readable medium or a machine-readable medium, such as a memory 804, a storage device 806, or a memory on a processor 802.

[0134] The high-speed controller 808 manages bandwidth-intensive operations of the computing device 800, while the low-speed controller 812 manages less bandwidth-intensive operations. This allocation of functions is merely exemplary. In one embodiment, the high-speed controller 808 is coupled to the memory 804, the display 816 (e.g., through a graphics processor or accelerator), and is connected to the high-speed expansion port 810, which can accommodate various expansion cards (not shown). The low-speed controller 812 can be coupled to the storage device 806 and the low-speed expansion port 814. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device (e.g., a switch or router), for example, through a network adapter.

[0135] As shown in the drawings, computing device 800 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 820, or multiple times in a group of such servers. In addition, it can be implemented as part of a rack server system 824. In addition, it can be implemented in a personal computer (such as a laptop computer 822). Alternatively, components from computing device 800 can be combined with other components (not shown) in a mobile device, such as device 850. Each of these devices can contain one or more computing devices 800, 850, and the entire system can be composed of multiple computing devices 800, 850 that communicate with each other.

[0136] Computing device 850 includes, among other components, a processor 852, a memory 864, an input / output device (e.g., a display 854), a communication interface 866, and a transceiver 868. Device 850 may also be provided with a storage device, such as a micro drive or other device, to provide additional storage. Each of components 850, 852, 864, 854, 866, and 868 are interconnected using various buses, and several components may be mounted on a common motherboard or otherwise as appropriate.

[0137] The processor 852 can execute instructions in the computing device 850, including instructions stored in the memory 864. The processor can be implemented as a chipset of chips, which includes separate multiple analog and digital processors. The processor can provide cooperation with other components of the device 850, such as control of a user interface, applications running through the device 850, and wireless communications through the device 850.

[0138] The processor 852 can communicate with the user through the control interface 858 and the display interface 856 coupled to the display 854. The display 854 can be, for example, a TFTLCD (thin film transistor liquid crystal display) display or an OLED (organic light emitting diode) display or other appropriate display technology. The display interface 856 may include appropriate circuits for driving the display 854 to present graphics and other information to the user. The control interface 858 can receive commands from the user and convert them to submit to the processor 852. In addition, the external interface 862 can provide communication with the processor 852, thereby realizing near-field communication of the device 850 with other devices. For example, in some embodiments, the external interface 862 can provide wired communication or wireless communication, and multiple interfaces can be used.

[0139] The memory 864 stores information in the computing device 850. The memory 864 can be implemented as one or more of a computer-readable medium or multiple computer-readable media, a volatile memory unit or multiple volatile memory units, or a non-volatile memory unit or multiple non-volatile memory units. In addition, an expansion memory 884 can be provided and connected to the device 850 through an expansion interface 882, which can include, for example, a SIMM (single in-line memory module) card interface. Such an expansion memory 884 can provide additional storage space for the device 850, and can also store applications or other information for the device 850. Specifically, the expansion memory 884 can include instructions for performing or supplementing the above-mentioned processing, and can also include security information. Therefore, for example, the expansion memory 884 can be a security module for the device 850 and can be programmed by instructions that allow secure use of the device 850. In addition, security applications can be provided via a SIMM card together with additional information, such as placing identification information into the SIMM card in a non-hackable manner.

[0140] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one embodiment, a computer program product may be tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods (such as the above methods). The information carrier may be a computer-readable medium or a machine-readable medium, such as memory 864, expansion memory 884, or memory on processor 852, which may be received, for example, via transceiver 868 or external interface 862.

[0141] The device 850 can communicate wirelessly via a communication interface 866, which may include digital signal processing circuitry when necessary. The communication interface 866 can provide communications under various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messages, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, etc. Such communications can be performed, for example, via a radio frequency transceiver 868. In addition, short-range communications can be performed, such as using Bluetooth, WiFi or other such transceivers (not shown). In addition, a GPS (global positioning system) receiver module 880 can provide additional navigation and location-related wireless data to the device 850, which can be used as appropriate by an application running on the device 850.

[0142] The device 850 may also communicate in an auditory manner using an audio codec 860, which may receive verbal information from a user and convert it into usable digital information. Likewise, the audio codec 860 may generate audible sounds for the user, for example, through a speaker, such as in a headset of the device 850. Such sounds may include sounds from a voice phone call, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on the device 850.

[0143] As shown in the drawings, the computing device 850 can be implemented in many different forms. For example, it can be implemented as a cellular phone 880. It can also be implemented as part of a smart phone 882, a personal digital assistant, or other similar mobile devices.

[0144] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system comprising at least one programmable processor, which can be special purpose or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0145] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0146] To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including sound, voice, or tactile input.

[0147] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser, through which a user can interact with embodiments of the systems and techniques described herein), or any combination of these back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0148] A computing system may include clients and servers. Clients and servers are generally remote from each other and interact through a communication network. The relationship between client and server is generated by computer programs running on the respective computers and having a client-server relationship to each other.

[0149] In some embodiments, Figure 8 The computing device shown may include sensors that interface with a virtual reality headset (VR headset / HMD device 890). For example, computing device 850 or Figure 8 One or more sensors included on the other computing devices shown may provide input to the VR headset 890 or, in general, to the VR space. Sensors may include, but are not limited to, touch screens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device 850 may use sensors to determine the absolute position and / or detected rotation of the computing device in the VR space, which may then be used as input to the VR space. For example, the computing device 850 may be incorporated into the VR space as a virtual object, such as a controller, laser pointer, keyboard, weapon, etc. When incorporated into the VR space, positioning the computing device / virtual object by the user may allow the user to position the computing device to view the virtual object in some manner in the VR space.

[0150] In some embodiments, one or more input devices included or connected to the computing device 850 may be used as input to the VR space. The input device may include, but is not limited to, a touch screen, a keyboard, one or more buttons, a touch pad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, headphones or earbuds with input functions, a game controller, or other connectable input devices. When the computing device is incorporated into the VR space, a user interacting with the input device included on the computing device 850 may cause specific actions to occur in the VR space.

[0151] In some embodiments, one or more output devices included on the computing device 850 may provide output and / or feedback to a user of the VR headset 890 in the VR space. The output and feedback may be visual, tactile, or audio. The output and / or feedback may include, but are not limited to, rendering the VR space or virtual environment, vibrating, turning on and off or blinking and / or flashing one or more lights or flashes, sounding an alarm, playing a ringtone, playing a song, and playing an audio file. The output devices may include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light emitting diodes (LEDs), flashes, and speakers.

[0152] In some embodiments, the computing device 850 may be placed within a VR headset 890 to create a VR system. The VR headset 890 may include one or more positioning elements that allow the computing device 850, such as a smartphone 882, to be placed in an appropriate position within the VR headset 890. In such embodiments, the display of the smartphone 882 may present a stereoscopic image representing a VR space or virtual environment.

[0153] In some embodiments, computing device 850 may appear as another object in a computer-generated 3D environment. User interactions with computing device 850 (e.g., rotation, shaking, touching a touch screen, sliding a finger across a touch screen) may be interpreted as interactions with objects in the VR space. As just one example, the computing device may be a laser pointer. In such an example, computing device 850 appears as a virtual laser pointer in a computer-generated 3D environment. As the user manipulates computing device 850, the user in the VR space sees the movement of the laser pointer. The user receives feedback from the interaction with computing device 850 in the VR environment on computing device 850 or VR headset 890.

[0154] In some embodiments, the computing device 850 may include a touch screen. For example, a user may interact with the touch screen in a manner that may mimic what is happening on the touch screen and what is happening in the VR space. For example, a user may use a pinching motion to zoom in on the content displayed on the touch screen. This pinching motion on the touch screen causes the information provided in the VR space to be magnified. In another example, the computing device may be presented as a virtual book in a computer-generated 3D environment. In the VR space, pages of a book may be displayed in the VR space, and a user's finger sliding across the touch screen may be interpreted as turning / flipping pages of a virtual book. As each page is turned / flipped, in addition to seeing the page content change, audio feedback may also be provided to the user, such as the sound of turning pages in a book.

[0155] In some embodiments, one or more input devices other than the computing device (e.g., a mouse, a keyboard) may be rendered in the computer-generated 3D environment. The rendered input devices (e.g., a rendered mouse, a rendered keyboard) may be used to render in the VR space to control objects in the VR space.

[0156] Computing device 800 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 850 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the disclosed embodiments.

[0157] In addition, the logic flow shown in the drawings does not require the specific order or sequential order shown to achieve the desired results. In addition, other steps can be provided to the illustrated flow, other steps can also be eliminated from the illustrated flow, and other components can be added to the system, and other components can also be deleted from the system. Therefore, other embodiments fall within the scope of the appended claims.

Claims

1. A computer-implemented method, the method being operated using an image processing system having at least one processing device, comprising: receiving a plurality of input images of participants in a three-dimensional video conference; generating a three-dimensional mesh proxy based on features extracted from the plurality of input images; generating, by a neural network, a neural texture based on the three-dimensional mesh proxy and the plurality of input images using inverse rendering in texture space; sampling at least a portion of the neural texture from the three-dimensional mesh proxy; as well as A composite image of the participant is generated based on the at least a portion that was sampled using a neural renderer.

2. The method according to claim 1, wherein: The features extracted include: a first set of features representing the participant's face; and A second set of features representing hair and a portion of a torso of the participant.

3. The method according to claim 1, wherein: The neural texture is a learned neural texture and the plurality of input images includes less than four captured image frames of the participant.

4. The method according to claim 1, wherein: Generating the neural texture comprises: Generate a latent representation for each input image; aggregating the representations generated for each input image into a vector; providing the vector to a texture generator network; and The neural texture representing the participant is received from the texture generator network.

5. The method according to claim 1, wherein: Generating the neural texture comprises: projecting the extracted features into a geometry-aware feature space based on the depth image and UV map associated with the input images, the projecting comprising generating a set of images having matching resolutions of the plurality of input images; Warping the set of images into UV space to generate a texture for each input image; generating, for each texture, a score map for each point in the texture space, the score map indicating a visibility metric in a corresponding one of the input images; Applying a soft-max operation to each score map to generate attention weights for each input image; and The textures are aggregated according to a weighted average, wherein the weighted average is determined according to the attention weights.

6. The method according to claim 1, further comprising: generating a representation of the participant representing the appearance of each surface point represented in the neural texture, and An appearance descriptor is stored for each surface point to represent high frequency details about the appearance of the participant.

7. The method according to any one of claims 1 to 6, wherein: The composite image is generated to depict the participant using a corrected eye gaze direction for a three-dimensional video conference that simulates a view of the participant looking at a stereoscopic display device associated with a system providing the three-dimensional video conference.

8. An image processing system, comprising: at least one processing device; as well as A memory storing instructions, which, when executed by the at least one processing device, cause the system to perform the following operations, including: receiving a plurality of input images of participants in a three-dimensional video conference; generating a three-dimensional mesh proxy based on features extracted from the plurality of input images; generating, by a neural network, a neural texture based on the three-dimensional mesh proxy and the plurality of input images using inverse rendering in texture space; generating a representation of the participant based at least in part on the three-dimensional mesh proxy and the neural texture; using the representation of the participant, sampling at least a portion of the neural texture from the three-dimensional mesh proxy; and In response to providing the at least a portion that was sampled to a neural renderer, a composite image of the participant is received from the neural renderer.

9. The system according to claim 8, wherein: The extracted features include: a first set of features representing the participant's face; and A second set of features representing hair and a portion of a torso of the participant.

10. The system according to claim 8, wherein: The neural texture is a learned neural texture and the plurality of input images includes less than four captured image frames of the participant.

11. The system according to claim 8, wherein: Generating the neural texture comprises: Generate a latent representation for each input image; aggregating the representations generated for each input image into a vector; providing the vector to a texture generator network; and The neural texture representing the participant is received from the texture generator network.

12. The system according to claim 8, wherein: Generating the neural texture comprises: projecting the extracted features into a geometry-aware feature space based on the depth image and UV map associated with the input images, the projecting comprising generating a set of images having matching resolutions of the plurality of input images; Warping the set of images into UV space to generate a texture for each input image; generating, for each texture, a score map for each point in the texture space, the score map indicating a visibility metric in a corresponding one of the input images; Applying a soft-max operation to each score map to generate attention weights for each input image; and The textures are aggregated according to a weighted average, wherein the weighted average is determined according to the attention weights.

13. A system according to any one of claims 8 to 12, wherein: The composite image is generated to depict the participant using a corrected eye gaze direction for a three-dimensional video conference that simulates a view of the participant looking at a stereoscopic display device associated with a system providing the three-dimensional video conference.

14. A non-transitory machine-readable medium having instructions stored thereon, wherein: The instructions, when executed by a processor, cause the computing device to: receiving a plurality of input images of participants in a three-dimensional video conference; generating a three-dimensional mesh proxy based on features extracted from the plurality of input images; generating, by a neural network, a neural texture based on the three-dimensional mesh proxy and the plurality of input images using inverse rendering in texture space; generating a representation of the participant based at least in part on the three-dimensional mesh proxy and the neural texture; using the representation of the participant, sampling at least a portion of the neural texture from the three-dimensional mesh proxy; providing the at least a portion of the sample to a neural renderer; as well as A composite image of the participant is received, the composite image of the participant representing a previously unseen view of the participant.

15. The machine-readable medium of claim 14, wherein: The extracted features include: a first set of features representing the participant's face; and A second set of features representing the participant's hair and a portion of her torso.

16. The machine-readable medium of claim 14, wherein: The neural texture is a learned neural texture generated using the plurality of input images, the plurality of input images comprising less than four captured image frames of the participant.

17. The machine-readable medium of claim 14, wherein: Generating the neural texture comprises: Generate a latent representation for each input image; aggregating the representations generated for each input image into a vector; providing the vector to a texture generator network; and The neural texture representing the participant is received from the texture generator network.

18. The machine-readable medium of claim 14, wherein: Generating the neural texture comprises: projecting the extracted features into a geometry-aware feature space based on the depth image and UV map associated with the input images, the projecting comprising generating a set of images having matching resolutions of the plurality of input images; Warping the set of images into UV space to generate a texture for each input image; generating, for each texture, a score map for each point in the texture space, the score map indicating a visibility metric in a corresponding one of the input images; Applying a soft-max operation to each score map to generate attention weights for each input image; and The textures are aggregated according to a weighted average, wherein the weighted average is determined according to the attention weights.

19. The machine-readable medium of claim 14, wherein: The generated representation of the participant represents the appearance of each surface point represented in the neural texture, and wherein the instructions further include storing an appearance descriptor for each surface point to represent high frequency details about the appearance of the participant.

20. The machine-readable medium of any one of claims 14 to 19, wherein: The composite image is generated to depict the participant using a corrected eye gaze direction for a three-dimensional video conference that simulates a view of the participant looking at a stereoscopic display device associated with a system providing the three-dimensional video conference.

Images