Color and infrared 3D reconstruction using implicit radiance functions

Abstract

The image is rendered based on a Neural Radiance Field (NeRF) volumetric representation of the scene, where the NeRF representation is based on captured frames of video data, where each frame includes a color image of the scene, a wide field IR image, and multiple depth IR images. Each depth IR image is captured when the scene is illuminated by a different pattern of IR light points, where the illumination by the patterns occurs at different times. The NeRF representation provides a mapping between positions and viewing directions to colors and optical densities at each position in the scene, where the colors and optical densities at each position allow the scene to be viewed from a new viewpoint, where the NeRF representation provides a mapping between positions and viewing directions to IR values ​​for each of the different patterns of IR light points from the new viewpoint.

Description

[Technical field]

[0001] Field Embodiments relate to capturing and rendering three-dimensional (3D) video. Embodiments further relate to using Neural Radiance Fields (NeRFs) to train neural network models for use in re-rendering images for display. [Background technology]

[0002] background The rise of Augmented Reality (AR) and Virtual Reality (VR) has created a demand for high-quality display of 3D content (e.g., humans, characters, actors, animals, and / or the like) using performance capture rigs (e.g., camera rigs and video rigs). Recently, real-time performance capture systems have enabled new use cases for telepresence, augmented video, and live performance broadcasting (in addition to offline multi-view performance capture systems), many of which provide adaptive computer-generated views of a scene depending on the observer's position and viewpoint. Summary of the Invention [Problem to be solved by the invention]

[0003] Some computers configured to render computer graphic objects can render the object in a specified view given multiple existing views. For example, given several depth and color images captured from a camera of a scene containing such a computer graphic object, it may result in synthesizing a new view of the scene from a different perspective. The scene may be either a real scene where the view is captured using physical color and depth sensors, or a synthetic scene where the view is captured using a rendering algorithm such as rasterization. For real scenes, many depth sensing technologies exist, such as time-of-flight sensors, structured light-based sensors, stereo or multi-view stereo algorithms. Such technologies may include visible or infrared sensors with passive or active illumination patterns, which may vary in time.

[0004] Existing performance capture systems may suffer from one or more technical issues, including some combination of distorted geometry, poor texturing, and inaccurate lighting, poorly resolved edges, and inaccurate colors, and therefore may have difficulty reaching the quality levels required for AR and VR applications. These technical issues may result in a less than desirable final user experience. [Means for solving the problem]

[0005] overview In at least one embodiment, a method includes receiving an image rendered based on a Neural Radiance Field (NeRF) volumetric representation of a scene, where the NeRF representation of the scene is based on captured frames of video data, each captured frame including a color image of the scene, a wide-field IR image of the scene captured when the scene is illuminated by a wide-field IR light source, and a plurality of depth IR images of the scene. Each of the depth IR images is captured when the scene is illuminated by a different pattern of IR light points, where the illumination by the different patterns of IR light points occurs at different times. The NeRF representation provides a mapping between a position and a viewing direction to a color and optical density at each position in the scene, where the color and optical density at each position in the scene allows the scene to be viewed from a new viewpoint, and the NeRF representation provides a mapping between a position and a viewing direction to an IR value for each of the different patterns of IR light points from the new viewpoint. A synthesis function is defined for generating an enhanced image from the received images, and the synthesis function is calculated using the trained neural network based on minimizing a loss function between a predicted image generated by the neural network and a ground truth image captured by the ground truth camera during training.

[0006] Implementations may include one or more of the following features, either alone or in any combination with each other.

[0007] The method may further include, prior to the step of receiving the image, capturing a 3D model of the scene using a volumetric capture system, and rendering the image based on the NeRF volumetric representation.

[0008] Both the ground truth camera and the volume capture system can be pointed at the view during training, with the ground truth camera producing higher quality images than the volume capture system.

[0009] The NeRF representation of a scene can be configured to generate position- and viewpoint-specific color, IR, and density image data of the scene for multiple viewing positions and viewing directions relative to the scene.

[0010] The location-specific and viewpoint-specific IR image data can include wide-field IR image data and patterned IR image data for each different pattern of IR light spots.

[0011] The loss function may include a reconstruction loss based on a reconstruction difference between a segmented ground truth image mapped to activations of a layer in the neural network and a segmented predicted image mapped to activations of a layer in the neural network, where the segmented ground truth image is segmented by a ground truth segmentation mask to remove background pixels, where the segmented predicted image is segmented by a predicted segmentation mask to remove background pixels, where the predicted segmentation mask is predicted based on a combination of both visible light information captured for the frame and IR light captured for the frame.

[0012] The reconstruction difference may be saliency reweighted to downweight the reconstruction difference in pixels above a maximum error or below a minimum error.

[0013] The loss function can include a mask loss based on a mask difference between a ground truth segmentation mask and a predicted segmentation mask, where the predicted segmentation mask is predicted based on a combination of both visible light information captured for the frame and IR light captured for the frame.

[0014] The predicted image may be one of a predicted stereo pair of images, and the loss function may include a stereo loss based on a stereo difference between the predicted stereo pair of images.

[0015] The neural network can be based on a fully convolutional model. The step of calculating the composite function using the neural network may include calculating a composite function at a left eye viewpoint and calculating a composite function at a right eye viewpoint.

[0016] The step of calculating the composite function using a neural network may be performed in real time.

[0017] A color image of a scene in a frame can include demosaiced color channels.

[0018] A color image of a scene in a frame can include three independent, non-demosaiced color channels, and a captured frame of video data underlying the NeRF representation of the scene includes three independent, non-demosaiced color channels.

[0019] In another general aspect, a performance capture system includes a volume capture system configured to render at least one image reconstructed from at least one viewpoint of a captured 3D model, the at least one image including imperfections, and a rendering system configured to receive the at least one image from the volume capture system and generate in real time at least one enhanced image in which imperfections in the at least one received image are reduced. The rendering system includes a neural network configured to generate the at least one enhanced image by training prior to use, the training including minimizing a loss function between predicted images generated by the neural network during training and corresponding ground truth images captured by at least one ground truth camera calibrated with the volume capture system during training, the minimization of the loss function being based on captured frames of video data, each captured frame including a color image of a scene, a wide field IR image of the scene captured when the scene is illuminated by a wide field IR light source, and a plurality of depth IR images of the scene, each of the depth IR images captured when the scene is illuminated by a different pattern of IR light points.

[0020] Example embodiments will become more fully understood from the following detailed description and the accompanying drawings provided herein, in which like elements are represented with like reference numerals, and which are provided for purposes of illustration only and therefore not as limitations of example embodiments. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a block diagram of a performance capture system in accordance with at least one exemplary embodiment. [Figure 2A] FIG. 1 illustrates a block diagram of a rendering system in accordance with at least one exemplary embodiment. [Figure 2B] FIG. 1 is a block diagram of an example of a 3D content system according to at least one exemplary embodiment. [Figure 3A] FIG. 2 illustrates a method for rendering frames of 3D video according to at least one exemplary embodiment. [Figure 3B] FIG. 2 illustrates a method for rendering frames of 3D video according to at least one exemplary embodiment. [Figure 4] FIG. 1 is a block diagram of a learning module system in accordance with at least one exemplary embodiment. [Diagram 5] FIG. 13 is a block diagram of a neural re-rendering module according to at least one exemplary embodiment. [Figure 6A] FIG. 1 illustrates layers of a convolutional neural network without sparsity constraint. [Figure 6B] FIG. 1 illustrates layers of a convolutional neural network with sparsity constraint. [Figure 7A] FIG. 1 illustrates a deep learning technique for generating a visually enhanced re-rendered image from a lower quality image in accordance with at least one exemplary embodiment. [Figure 7B] FIG. 1 illustrates a deep learning technique for generating a visually enhanced re-rendered image from a lower quality image in accordance with at least one exemplary embodiment. [Figure 8] FIG. 2 is a diagram showing an example of a low-quality image. [Figure 9] FIG. 1 illustrates exemplary training data for a convolutional neural network model in accordance with at least one exemplary embodiment. [Figure 10A] FIG. 1 illustrates a diagram showing reconstruction loss in accordance with at least one exemplary embodiment. [Figure 10B] FIG. 1 illustrates a mask loss in accordance with at least one exemplary embodiment. [Figure 10C] FIG. 1 illustrates a diagram showing head loss in accordance with at least one exemplary embodiment. [Figure 10D] FIG. 1 illustrates stereo loss in accordance with at least one exemplary embodiment. [Figure 10E]FIG. 1 illustrates a diagram showing time loss in accordance with at least one exemplary embodiment. [Figure 10F] FIG. 1 illustrates a diagram showing saliency loss in accordance with at least one exemplary embodiment. [Figure 11] 1 illustrates an example of a computing device and a mobile computing device in accordance with at least one exemplary embodiment. [Figure 12] 1 is a block diagram of an exemplary output image for providing content on a stereoscopic display in accordance with at least one exemplary embodiment. [Figure 13] 13 is a schematic flow diagram of a process 1300 for implementing the techniques described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] It should be noted that these figures are intended to illustrate general features of methods, structures, and / or materials utilized in certain exemplary embodiments and to supplement the written description provided below. However, these figures are not to scale, may not precisely reflect the exact structural or performance characteristics of a given embodiment, and should not be construed as defining or limiting the range of values ​​or properties encompassed by the exemplary embodiments. For example, relative thicknesses and positions of layers, regions, and / or structural elements may be reduced or exaggerated for clarity. The use of similar or identical reference numbers in various figures is intended to indicate the presence of similar or identical elements or features.

[0023] Detailed Description of the Preferred Embodiments While exemplary embodiments may include various modifications and alternative forms, such embodiments are shown by way of example in the drawings and described in detail herein. It should be understood, however, that there is no intention to limit the exemplary embodiments to the particular forms disclosed, but on the contrary, the exemplary embodiments are intended to cover all modifications, equivalents, and alternatives falling within the scope of the claims. Like numbers refer to like elements throughout the description of the drawings.

[0024] A traditional approach to synthesizing new views of a scene from existing views uses a neural radiance field (NeRF), in which a static scene can be represented as a continuous 5-dimensional function that outputs the radiance emitted in each direction (θ,φ) at each point (x,y,z) in space, and a density at each point that acts like a differential opacity that controls the amount of radiance accumulated by a ray passing through (x,y,z). This approach optimizes a deep fully connected neural network (MLP) without convolutional layers to represent the 5-dimensional function by regressing from a single 5D coordinate (x,y,z,θ,φ) to a single volumetric density and a view-dependent RGB color.

[0025] To render this 5-dimensional function, i.e., NeRF, one can 1) march a camera ray through the scene to generate a set of sampled 3D points, 2) use those points and their corresponding 2D observation directions as inputs to a neural network to generate an output set of colors and densities, and 3) accumulate those colors and densities into a 2D image using classical volumetric rendering techniques. Because this process is necessarily differentiable, one can use gradient descent to optimize the NeRF by minimizing the error between each observed image and the corresponding view rendered from the display. Minimizing this error across multiple views encourages the network to predict a consistent model of the scene by assigning high volumetric density and accurate colors to locations that contain the actual underlying scene content.

[0026] NeRF is a continuous volumetric representation. It is a function F:(x,d)→(c,σ) that maps a 3D position x=(x,y,z) and a viewing direction d=(φ,θ) to an RGB color c=(r,g,b) and density σ. When combined with volumetric rendering techniques, NeRF can represent a scene with photorealistic quality. For this reason, NeRF is built to tackle the problem of photorealistic human capture.

[0027] Although NeRF has been very successful, its attempt to represent human-occupied scenes with lifelike quality can result in several imperfections: for example, scene features can be rendered with distorted geometry, poor texturing, inaccurate lighting, poorly resolved edges, inaccurate colors, etc.

[0028] To address these limitations, machine learning techniques and models can be used to improve the performance of NeRF. For example, high-quality training data can be generated and used to train a machine learning model that is used to improve the rendering of high-quality images from low-quality images based on NeRF. In some implementations, the training data can include a wide-field infrared image of the scene, in addition to a color (e.g., RGB) image of the scene, and texture data based on multiple different infrared patterns overlaid on objects in the scene. The infrared wide-field image and infrared texture data can be used in the machine learning model to generate a NeRF for the scene.

[0029] A performance capture rig (i.e., a performance capture system) may be used to capture a subject (e.g., a person) and its movements in three dimensions (3D). The performance capture rig may include a volumetric capture system configured to capture data necessary to generate a 3D model (e.g., NeRF) and (in some cases) render a 3D volumetric reconstruction (i.e., an image) using a volumetric reconstruction of the views. A variety of volumetric capture systems may be implemented, including (but not limited to) active stereo cameras, time-of-flight (TOF) systems, lidar systems, passive stereo cameras, and the like. In some implementations, a single volumetric capture system may be used, while in other embodiments, multiple volumetric capture systems may be used (e.g., in coordinated capture).

[0030] The volumetric reconstruction can render a video stream of images (e.g., in real time) and can render separate images corresponding to the left and right eye viewpoints. The 2D images of the left and right eye viewpoints can be displayed on a stereo display. The stereo display can be a head-tracking stereo display. A variety of stereo displays can be implemented, including, but not limited to, an augmented reality (AR) glasses display, a virtual reality (VR) headset display, and an autostereo display (e.g., a head-tracking autostereo display). In some implementations, the stereo display can be used in a telepresence system that can present a highly realistic rendering of a person at a first location to a person at a second location.

[0031] Imperfections (i.e., artifacts) may be present in the rendered 2D images and / or their display on a stereo display. The artifacts may include graphic artifacts such as intensity noise, low-resolution textures, off-color, etc. The artifacts may further include stereo artifacts such as inconsistent left and right views. The artifacts may be due to limitations / issues associated with the performance capture rig. For example, complexity and cost constraints may limit the data that the performance capture rig can collect. Also, the artifacts may be due to limitations / issues associated with collecting and transmitting depth information to enable a 3D image to be rendered. Additionally, the artifacts may be due to limitations (e.g., bandwidth) associated with data transfer over a network. This disclosure describes systems and methods that reduce or eliminate artifacts regardless of the source of the artifacts. Thus, the disclosed systems and methods are not limited to any particular performance capture system or stereo display.

[0032] In one possible implementation, technical issues associated with existing performance capture systems can result in 3D volumetric reconstruction images that contain holes, noise, low-resolution textures, and color artifacts. These technical issues can result in an undesirable user experience in VR and AR applications.

[0033] A technical solution to the aforementioned technical problem implements machine learning to enhance volumetric video in real time. A geometric non-rigid reconstruction pipeline can be combined with deep learning to produce higher quality images. The disclosed system can focus on visually salient regions (e.g., human faces) and discard irrelevant information such as backgrounds where the background is precisely defined, and can generate high quality images using multispectral information including visible light (color) and infrared (IR) light information collected from the scene. The described solution can generate temporally stable renderings for implementation in VR and AR applications where left and right views need to be consistent for an optimal user experience.

[0034] Technical solutions can include real-time performance capture (i.e., image and / or video capture) to obtain approximate geometry and texture in real-time. The final 2D rendering output of such systems can be of low quality due to geometric artifacts, poor texturing, and inaccurate lighting. Thus, implementations can use deep learning to enhance the final rendering to achieve high-quality results in real-time. For example, a deep learning architecture (see Figures 7A and 7B) can be used that takes as input the final 2D rendering images from a deferred shading deep buffer and / or multi-view performance capture system and learns to enhance such images in real-time to produce a final high-quality re-rendering. This approach is referred to as neural re-rendering.

[0035] Described herein is a neural re-rendering technique. Technical advantages of using the neural re-rendering technique include the ability to learn how to enhance in real time low quality output from a performance capture system where the image contains holes, noise, low resolution textures, and color artifacts. Some examples of low quality images are shown in FIG. 8. Furthermore, a binary segmentation mask that separates the user from the remaining background can be predicted. Technical advantages of using the neural re-rendering technique also include a way to reduce the overall bandwidth and computational effort required for such deep architectures by having the network learn a mapping from low resolution input images to high resolution output renderings during the training phase, and then using low resolution images (e.g., enhanced images) from the live performance capture system.

[0036] The technical advantages of using neural re-rendering techniques also include a specialized loss function that can use semantic information to produce high-quality results on faces. To reduce the impact of outliers, a saliency reweighting scheme can be used that focuses the loss on the most relevant regions. The loss function is designed for VR and AR headsets, and its goal is to predict two consistent views of the same object. The technical advantages of using neural re-rendering techniques also include the ability to enable temporally stable re-rendering by enforcing consistency between successively reconstructed frames.

[0037] FIG. 1 illustrates a block diagram of a performance capture system (i.e., capture system) according to at least one exemplary embodiment. As illustrated in FIG. 1, capture system 100 includes a 3D camera rig with witness cameras 110, an encoder 120, a decoder 130, a rendering module 140, and a learning module 150. Camera rig with witness cameras 110 includes a first camera set used to capture 3D video as video data 5, and at least one witness camera used to capture high quality (e.g., compared to the first camera set) images from at least one perspective as ground truth image data 30. The camera rig may also include an infrared sensor for capturing infrared (IR) light that may be used, for example, to determine depth and / or texture information and / or to distinguish objects or subjects in a scene from a background. Ground truth image data 30 may be images that include more detailed images (e.g., higher definition, higher resolution, higher pixel count, adding more / better depth information, etc.) and / or post-capture processing to improve image quality, as compared to frames or images associated with 3D video. The ground truth image data 30 may include (a set of) ground truth images, image labels, image segmentation information, image and / or segment classification information, location information, etc. The ground truth image data 30 is used by the learning module 150 to train a neural network model. Each image in the ground truth image data 30 may have a corresponding frame of the video data 5.

[0038] The encoder 120 may be configured to compress the 3D video captured by the first set of cameras. The encoder 120 may be configured to receive the video data 5 and generate compressed video data 10 using a standard compression technique. The decoder 130 may be configured to receive the compressed video data 10 and generate reconstructed video data 15 using the inverse of the standard compression technique. The dashed and dotted lines shown in FIG. 1 indicate that in an alternative implementation, the encoder 120 and the decoder 130 may be bypassed and the video data 5 may be input directly to the rendering module 140. This may reduce the processing resources used by the capture system 100. However, the learning module 150 may not include errors introduced by compression and decompression in the training process.

[0039] The rendering module 140 is configured to generate a left eye view 20 and a right eye view 25 based on the reconstructed video data 15 (or the video data 5). The left eye view 20 may be an image displayed on a left eye display of a head mounted display (HMD). The right eye view 25 may be an image displayed on a right eye display of the HMD. The rendering may include processing a 3D model of a scene (e.g., a NeRF of a scene) associated with the reconstructed video data 15 (or the video data 5) to generate a digital image. The 3D model may include, for example, shading information, lighting information, texture information, geometric information, etc. The rendering may include implementing a rendering algorithm by a graphical processing unit (GPU). Thus, the rendering may include passing the 3D model to the GPU.

[0040] The learning module 150 may be configured to train a neural network or model to generate a high quality image based on one or more low quality images, which may include images captured at different wavelength ranges, which may include the visible and IR wavelength ranges. In an exemplary implementation, an image is iteratively predicted based on the left eye view 20 (or right eye view 25) using the neural network or model. Each iteration of the predicted image is then compared to a corresponding image selected from the ground truth image data 30 using a loss function until the loss function is minimized (or below a threshold). The learning module 150 is described in more detail below.

[0041] 2A illustrates a block diagram of a rendering system according to at least one exemplary embodiment. As shown in FIG. 2A, the rendering system 200 includes a decoder 130, a rendering module 140, and a neural re-rendering module 210. As shown in FIG. 2A, the compressed video data 10 is decompressed by the decoder 130 to generate reconstructed video data 15. Then, the rendering module 140 generates a left eye view 20 and a right eye view 25 based on the reconstructed video data 15.

[0042] The neural re-rendering module 210 is configured to generate a re-rendered left eye view 35 based on the left eye view 20 and a re-rendered right eye view 40 based on the right eye view 25. The neural re-rendering module 210 is configured to generate the re-rendered left eye view 35 as a higher quality representation of the left eye view 20 using a neural network or model trained by the learning module 150. The neural re-rendering module 210 is configured to generate the re-rendered right eye view 40 as a higher quality representation of the right eye view 25 using a neural network or model trained by the learning module 150. The neural re-rendering module 210 is described in more detail below.

[0043] The capture system 100 shown in FIG. 1 may be a first stage (or stage 1) of an enhanced video rendering technique, and the rendering system 200 shown in FIG. 2A may be a second stage (or stage 2) of the enhanced video rendering technique. FIG. 3A (stage 1) and FIG. 3B (stage 2) illustrate a method of rendering frames of a 3D video according to at least one exemplary embodiment. The steps described with respect to FIG. 3A and FIG. 3B may be performed by execution of software code stored in a memory associated with the device and / or service (e.g., a cloud computing service) and executed by at least one processor associated with the device and / or service. However, alternative embodiments such as a system implemented as a dedicated processor are also contemplated. Although the steps described below are described as being performed by a processor, each step does not necessarily have to be performed by the same processor. In other words, at least one processor may perform the steps described below with respect to FIG. 3A and FIG. 3B.

[0044] 2B illustrates a block diagram of an example of a 3D content system according to at least one exemplary embodiment. The 3D content system 201 can be used by multiple people. Here, the 3D content system 201 is used by a first person 202 and a second person 204. For example, the first person 202 and the second person 204 are using the 3D content system 201 to participate in a 3D telepresence session. In such an example, the 3D content system 201 enables each of the people 202, 204 to see a highly realistic and visually consistent representation of each other, thereby facilitating them to interact with each other as if they were physically present with each other.

[0045] Each of the people 202, 204 can have a corresponding 3D pod, where the first person 202 has a pod 206 and the second person 204 has a pod 208. The pods 206 and 208 can provide functionality related to the 3D content, including, but not limited to, capturing images for 3D viewing, processing and presenting image information, and processing and presenting audio information. The pods 206 and / or 208 can include a collection of processors and sensing devices integrated as a single unit.

[0046] The 3D content system 201 can include one or more 3D displays, where 3D display 210 is provided in pod 206 and 3D display 212 is provided in pod 208. The 3D displays 210 and / or 212 can provide stereoscopic viewing to a respective observer (here, e.g., person 202 or 204) using any of several types of 3D display technology. In some implementations, the 3D displays 210 and / or 212 can include standalone units (e.g., freestanding or wall-hung units). In some implementations, the 3D displays 210 and / or 212 can include wearable technology (e.g., head-mounted displays). In some embodiments, the 3D displays 210 and / or 212 can include autostereoscopic display assemblies.

[0047] The 3D content system 201 can be connected to one or more networks, where network 214 is connected to pod 206 and pod 208. Network 214 can be a publicly available network (e.g., the Internet) or a private network, to name two examples. Network 214 can be wired, wireless, or a combination of the two. Network 214 can include or utilize one or more other devices or systems, including, but not limited to, one or more servers (not shown).

[0048] The pods 206 and / or 208 may include multiple components related to capturing, processing, transmitting or receiving 3D information, and / or presenting 3D content. The pods 206 and 208 may include one or more illumination sources for illuminating the objects in the scene (e.g., the first and second persons 202, 204) with visible and / or IR light, and a camera for capturing image content to generate images included in the 3D presentation. In some implementations, the illumination source may include an IR laser that shines through a diffractive optical element (DOE) and generates a pattern of IR laser light spots that illuminate the scene to be imaged. The pattern of IR laser light spots may be generated periodically with a period synchronized with the frame rate at which the visible image of the scene is captured, so that an IR image of the pattern of IR laser light spots may be captured within 1 / 20th of a second of capturing a color image of the scene. The IR laser light spots may be used to determine and assign depth information within the scene from the captured images. In some implementations, multiple distinct (e.g., non-degenerate) patterns of laser light spots can be generated and used to illuminate a scene, with each distinct pattern being generated periodically with a period synchronized with the frame rate at which a visible image of the scene is captured, such that an IR image of each of the distinct patterns of IR laser light spots can be acquired within 1 / 20th of a second of acquiring a color image of the scene. Using multiple patterns allows the richer information from the multiple patterns to be used to generate better depth data for objects in the scene.

[0049] Also, the scene can be illuminated with a wide field of view, unpatterned IR light from one or more IR illumination sources that provide a substantially uniform IR light intensity over a wide angle, and an IR image of the scene can be acquired. The wide field of view, unpatterned IR light from one or more illumination sources can be generated periodically with a period synchronized with the frame rate at which the visible image of the scene is captured, and an IR image of the scene (i.e., a "guide" image) can be captured while the scene is illuminated with the wide field of view, unpatterned IR light, such that the wide field of view IR image of the scene can be acquired within 1 / 20th of a second of acquiring the color image of the scene. The environment in which the image of the subject 202, 204 is captured can include a background 240, 242 that has a high reflectivity to IR light, such that the IR image of the subject 202, 204 can be used to distinguish the edges of the subject from the background in the captured image. The background 240, 242 can include a fabric material that includes a material that is highly reflective of IR light. For example, the fabric can include a metallic material (e.g., metallic fibers) woven into the fabric. In another example, the fibers of the textile material may be coated with an IR reflective pigment (IRP), which may include, for example, a metal oxide, a ceramic material, or an inorganic material.

[0050] Here, the pod 206 includes cameras 216 and 218, each of which can detect light in one or more wavelength ranges, including visible colors and IR wavelengths. For example, the cameras 216 and / or 218 can be essentially disposed within the housing of the pod 206, with the objective or lens of each camera 216 and / or 218 capturing image content through one or more openings in the housing. In some implementations, the cameras 216 and / or 218 can be separate from the housing, such as in the form of a stand-alone device (e.g., with a wired and / or wireless connection to the pod 206). The cameras 216 and 218 can be positioned and / or oriented to capture a sufficiently representative view (here) of the person 202. The cameras 216 and 218 preferably do not obstruct the view of the 3D display 210 to the person 202, although the placement of the cameras 216 and 218 can generally be selected arbitrarily. For example, one of cameras 216 and 218 can be located somewhere above the face of person 202 and the other can be located somewhere below the face. For example, one of cameras 216 and 218 can be located somewhere on the right side of the face of person 202 and the other can be located somewhere on the left side of the face. Pod 208 can include cameras 220 and 222 in a similar manner, for example.

[0051] The cameras 220 and 222 that detect visible light may include a light sensor array and color filters, allowing individual pixels of the sensor to detect different wavelengths of light. For example, each pixel includes sub-pixels for detecting red (R), green (G), and blue (B) wavelengths of light. The light signals detected by the sub-pixels of a pixel may be processed through a demosaicing process to assign a color to the pixel. In some implementations, the light signals detected by the sub-pixels are not processed through a demosaicing process before transmitting the separate R, G, and B channel data from one pod 206 to another pod 208 via the pipe neural re-rendering module 210. Rather, the separate R, G, and B channel data is processed by the neural re-rendering module 210 at the receiving pod to generate the image that is displayed.

[0052] The pods 206 and / or 208 may include one or more depth sensors for capturing depth data used in the 3D presentation. Such depth sensors (e.g., IR-sensitive cameras) may be considered part of the depth capture component of the 3D content system 201, which is used to characterize the scene captured by the pods 206 and / or 208 and accurately display it on the 3D display. The system may also track the position and orientation of the viewer's head, so that the 3D presentation can be rendered with an appearance that corresponds to the viewer's current viewpoint. Here, the pod 206 includes a depth sensor 224. Similarly, the pod 208 may include a depth sensor 224. Any of several types of depth sensing or depth capture may be used to generate the depth data. In some implementations, assisted stereo depth capture is performed. The scene may be illuminated using a pattern of IR light points that may be generated by a laser and a DOE included in the depth sensor 224, and stereo matching may be performed between the two respective cameras. Here, the depth sensor 224 operates, by way of example, using IR beams 228A and 228, which represent multiple beams generating a pattern of points. The beams 228A and 228B may travel, as the case may be, from the pod 206 toward a structure or other object (e.g., person 202) in the scene being 3D captured, and / or from such structure / object to a corresponding detector in the pod 206. The detected signals may be processed to generate depth data corresponding to a portion or the entirety of the scene. Thus, the beams 228A-B may be considered to relate to signals that the 3D content system 201 relies on to characterize the scene for purposes of 3D display. For example, the beams 228A-B may include IR signals. Similarly, the pod 208 may operate, by way of example, using beams 230A-B.

[0053] The depth data can include or be based on information about the scene that reflects the distance between a depth sensor (e.g., depth sensor 224) and an object in the scene. For content in an image that corresponds to an object in the scene, the depth data reflects the distance (or depth) to the object. For example, the spatial relationship between a camera and a depth sensor can be known and used to correlate an image from the camera with a signal from the depth sensor to generate depth data for the image.

[0054] In some implementations, depth capture can include approaches based on structured or coded light. For example, a pattern of IR light (e.g., a pattern of dots or stripes) can be distributed over the scene at a relatively high frame rate. A high frame rate is considered when, for example, the light signals are close enough together in time that the scene is not expected to change significantly between successive signals, even if people or objects are moving. The resulting pattern projected onto objects in the scene can be detected by an IR-sensitive camera in the depth sensor, and the detected pattern can be processed (e.g., using triangulation or other techniques) to determine the geometry of the scene at one or more points.

[0055] In some implementations, for each color image frame of video data of a scene captured by the cameras of the pods 206, 208, e.g., when capturing video data of a scene including a person 202, 204, at least one corresponding wide field IR image of the scene can be captured and multiple depth IR images can be captured, each of the multiple depth IR images capturing an IR image of a different pattern of IR points on the scene. The captured video data can then be used to train a machine learning model for each frame of the video data. The trained model can then be used to generate enhanced quality frames of video data from the low quality frames of video data based on the color image, the wide field IR image, and the multiple depth IR images of the scene captured for each frame of the low quality video data.

[0056] The images captured by the 3D content system 201 may be processed and then displayed as a 3D presentation, where the 3D image 204' is presented on a 3D display 210. Thus, the person 202 may perceive the 3D image 204' as a 3D representation of the person 204, who may be remote from the person 202. The 3D image 202' is presented on a 3D display 212. Thus, the person 204 may perceive the 3D image 202' as a 3D representation of the person 202. An example of three-dimensional information processing is described below.

[0057] The 3D content system 201 may enable participants (e.g., people 202 and 204) to engage in audio communication with each other and / or others. In some implementations, the pod 206 includes a speaker and a microphone (not shown). For example, the pod 208 may include a speaker and a microphone as well. Thus, the 3D content system 201 may enable people 202 and 204 to participate in a 3D telepresence session with each other and / or others.

[0058] FIG. 3A is a schematic flow chart of a process of training a machine learning model to generate a video image. As shown in FIG. 3A, in step S305, a camera rig including at least one witness camera is used to capture multiple frames of a first three-dimensional (3D) video. The capture of the frames of 3D video data includes capturing at least one color image of a scene, capturing at least one IR image of the scene when the scene is illuminated by a wide-field IR light source, and capturing multiple IR images of the scene, each IR image being captured when the scene is illuminated by a different pattern of IR light points. In some implementations, the captured color image of the scene may be a demosaiced image. In some implementations, the captured color image of the scene may include raw data (i.e., undemosaiced) of at least three independent color channels (e.g., red, green, and blue) for each pixel of the image. For example, a camera rig (e.g., a 3D camera rig with witness cameras 110) may include a first camera set used to capture 3D video (e.g., visible and IR image data) and at least one witness camera used to capture higher quality images (e.g., ground truth image data 30) (e.g., compared to the first camera set). The frames of the first 3D video may be video data captured by the first camera set.

[0059] In step S310, at least one two-dimensional (2D) ground truth image is captured for each of the plurality of frames of the first 3D video using at least one witness camera. For example, the at least one 2D ground truth image may be a high-quality image captured by the at least one witness camera. The at least one 2D ground truth image may be captured at substantially the same moment as a corresponding frame of the plurality of frames of the first 3D video.

[0060] In step S315, at least one of the plurality of frames of the first 3D video is compressed. For example, at least one of the plurality of frames of the first 3D video is compressed using a standard compression technique. In step S320, at least one of the plurality of frames of the first 3D video is decompressed. For example, at least one of the plurality of frames of the first 3D video is decompressed using a standard decompression technique corresponding to the standard compression technique.

[0061] In step S325, at least one first 2D left eye view image is rendered based on the decompressed frames, and at least one first 2D right eye view image is rendered based on the decompressed frames. For example, a 3D model of a scene corresponding to a frame of the decompressed first 3D video (e.g., reconstructed video data 15) is communicated to the GPU. The GPU can generate digital images (e.g., left eye view 20 and right eye view 25) based on the 3D model of the scene and return the digital images as the first 2D left eye view and the first 2D right eye view.

[0062] In step S330, a model of a left eye view of a head mounted display (HMD) is trained based on the rendered first 2D left eye view image and the corresponding 2D ground truth image, and a model of a right eye view of the HMD is trained based on the rendered first 2D right eye view image and the corresponding 2D ground truth image. The model is based on information in the visible channel as well as information in the IR channel (e.g., based on images of the scene captured under wide field of view and patterned IR illumination). For example, an image is iteratively predicted based on the first 2D left eye view using a neural network or model. Then, each iteration of the predicted image is compared to the corresponding 2D ground truth image using a loss function until the loss function is minimized (or falls below a threshold). Furthermore, an image is iteratively predicted based on the first 2D right eye view using a neural network or model. Then, each iteration of the predicted image is compared to the corresponding 2D ground truth image using a loss function until the loss function is minimized (or falls below a threshold).

[0063] FIG. 3B is a schematic flow chart of a process for generating and rendering an image based on compressed image data and a trained machine learning model. The compressed image data and model are based on information in the visible channel as well as information in the IR channel (e.g., based on images of a scene captured under wide field of view and patterned IR illumination). As shown in FIG. 3B, in step S335, compressed video data corresponding to a second 3D video is received. For example, video data captured using a 3D camera rig is captured, compressed, and communicated as a second 3D video at a remote device (e.g., by a computing device at a remote location). This compressed second 3D video is received by a local device. The second 3D video may be different from the first 3D video.

[0064] In step S340, the video data corresponding to the second 3D video is decompressed. For example, the second 3D video (e.g., the compressed video data 10) is decompressed using a standard decompression technique corresponding to the standard compression technique used by the remote device.

[0065] In step S345, a frame of the second 3D video is selected. For example, the next frame of the decompressed second 3D video can be selected for display on the HMD that plays the second 3D video. Alternatively or additionally, a buffer or queue of video frames can be utilized for playing the second 3D video. Thus, selecting a frame of the second 3D video can include selecting a frame from a queue based on a buffering or queuing technique (e.g., FIFO, LIFO, etc.).

[0066] In step S350, a second 2D left eye view image is rendered based on the selected frame, and a second 2D right eye view image is rendered based on the selected frame. For example, a 3D model of a scene corresponding to a frame of the decompressed second 3D video (e.g., reconstructed video data 15) is communicated to the GPU. The GPU can generate digital images (e.g., left eye view 20 and right eye view 25) based on the 3D model of the scene and return the digital images as the second 2D left eye view and the second 2D right eye view.

[0067] In step S355, the second 2D left eye view image is re-rendered using a convolutional neural network architecture and a trained model for the left eye view of the HMD, and the second 2D right eye view image is re-rendered using a convolutional neural network architecture and a trained model for the right eye view of the HMD. For example, the neural network or model trained in stage 1 may be used to generate a re-rendered second 2D left eye view (e.g., re-rendered left eye view 35) as a higher quality representation of the second 2D left eye view (e.g., left eye view 20). The neural network or model trained in stage 1 may be used to generate a re-rendered second 2D right eye view (e.g., re-rendered right eye view 35) as a higher quality representation of the second 2D right eye view (e.g., right eye view 25). Then, in step S360, the re-rendered second 2D left eye view image and the re-rendered second 2D right eye view image are displayed on at least one display of the HMD.

[0068] FIG. 4 illustrates a block diagram of a learning module system according to at least one exemplary embodiment. The learning module 150 can be or include at least one computing device and can represent virtually any computing device configured to perform the methods described herein. Thus, the learning module 150 can include various components that can be utilized to implement the techniques described herein, or different or future versions thereof. As an example, the learning module 150 is illustrated as including at least one processor 405 and at least one memory 410 (e.g., a non-transitory computer-readable medium).

[0069] As shown in FIG. 4, the learning module 150 includes at least one processor 405 and at least one memory 410. The at least one processor 405 and the at least one memory 410 are communicatively coupled via a bus 415. The at least one processor 405 may be utilized to execute instructions stored in the at least one memory 410, thereby implementing various features and functions described herein, or additional or alternative features and functions. The at least one processor 405 and the at least one memory 410 may be utilized for various other purposes. In particular, the at least one memory 410 may represent one example of various types of memory and associated hardware and software that may be used to implement any one of the modules described herein.

[0070] The at least one memory 410 may be configured to store data and / or information related to the learning module system 150. For example, the at least one memory 410 may be configured to store a model 420, a plurality of coefficients 425, and a plurality of loss functions 430. The at least one memory 410 further includes a metrics module 435 and an enumeration module 450. The metrics module 435 includes a plurality of error definitions 440 and an error calculator 445.

[0071] In an example implementation, the at least one memory 410 may be configured to store a code segment that, when executed by the at least one processor 405, causes the at least one processor 405 to select and communicate one or more of the plurality of coefficients 425. Additionally, the at least one memory 410 may be configured to store a code segment that, when executed by the at least one processor 405, causes the at least one processor 405 to receive information used by the learning module 150 system to generate new coefficients 425 and / or update existing coefficients 425. The at least one memory 410 may be configured to store a code segment that, when executed by the at least one processor 405, causes the at least one processor 405 to receive information used by the learning module 150 system to generate new models 420 and / or update existing models 420.

[0072] Model 420 represents at least one neural network model. A neural network model can define the operation of a neural network, the flow of operations, and / or the interconnections between operations. For example, operations can include normalization, padding, convolution, rounding, etc. A model can also define operations. For example, a convolution can be defined by a number of filters C, a spatial extent (or filter size) K×K, and a stride S. A convolution does not have to be square. For example, the spatial extent can be K×L. In the context of a convolutional neural network (see FIG. 6A and FIG. 6B), each neuron in the convolutional neural network can represent a filter. Thus, a convolutional neural network with 8 neurons per layer can have 8 filters using one (1) layer, 16 filters using two (2) layers, 24 filters using three (3) layers... 64 filters using 8 layers... 128 filters using 16 layers, etc. In a convolutional neural network, a layer can have any number of neurons.

[0073] Convolutional neural networks can have layers with different numbers of neurons. A K×K spatial extent (or filter size) can have K columns and K (or L) rows. The K×K spatial extent can be 2×2, 3×3, 4×4, 5×5, (K×L)2×4. The convolution involves centering the K×K spatial extent on a pixel, convolving all pixels in the spatial extent, and generating a new value for the pixel based on all (e.g., sum) of the convolutions of all pixels in the spatial extent. The spatial extent is then moved to a new pixel based on the stride, and the convolution is repeated for the new pixel. For example, the stride can be 1 or 2, where a stride of 1 moves to the next pixel and a stride of 2 skips the pixel.

[0074] The coefficients 425 represent variable values ​​that can be used in one or more of the model 420 and / or the loss function 430 to use and / or train the neural network. A unique combination of the model 420, the coefficients 425, and the loss function can define a neural network and a method for training a unique neural network. For example, a model of the model 420 can be defined to include two convolution operations and an interconnection between the two. The coefficients 425 can include corresponding entries that define the spatial range (e.g., 2×4, 2×2, etc.) and stride (e.g., 1, 2, etc.) of each convolution. Additionally, the loss function 430 can include corresponding entries that define the loss function and loss thresholds (e.g., minimum, maximum, minimum change, maximum change, etc.) for training the model.

[0075] The metrics module 435 includes multiple error definitions 440 and an error calculator 445. The error definitions may include, for example, a function or algorithm used to calculate an error and an error threshold (e.g., minimum, maximum, minimum change, maximum change, etc.). The error calculator 445 may be configured to use an algorithm to calculate an error between two images based on a pixel-by-pixel difference between the two images. Error types may include photometric error, peak signal-to-noise ratio (PSNR), structural similarity (SSIM), multi-scale SSIM (MS-SSIM), mean square error, perceptual error, etc. The enumeration module 450 may be configured to iterate over one or more of the coefficients 425.

[0076] In an exemplary implementation, one of the coefficients is modified by the enumeration module 450 for one of the models 420 while holding the rest of the coefficients constant. During each iteration (e.g., an iteration for training a left eye view), the processor 405 predicts an image using a model having a view (e.g., the left eye view 20) as an input and calculates a loss (possibly using the ground truth image data 30) until the loss function is minimized and / or the change in the loss is minimized. The error calculator 445 then calculates the error between the predicted image and the corresponding image of the ground truth image data 30. If the error is unacceptable (e.g., greater than a threshold or greater than a threshold change compared to the previous iteration), another coefficient is modified by the enumeration module 450. In an exemplary implementation, more than one loss function may be optimized. In this implementation, the enumeration module 450 may be configured to select from two or more loss functions.

[0077] According to an example implementation, from an image I (e.g., left eye view 20 and right eye view 25) rendered from a volumetric reconstruction (e.g., reconstructed video data 15), I e We can generate or compute an expanded version of I, denoted as I and I eThe transformation function between I and I can target telepresence, VR, or AR applications. Therefore, the following principles must be taken into account: a) users typically focus on salient features such as faces, so artifacts in those regions must be heavily penalized, b) when viewing in stereo, the network output must be consistent between left and right pairs to prevent user discomfort, and c) in VR applications, accurate segmentation masks are needed, since the renderings are composited into the virtual world. Furthermore, the enhanced image must be temporally consistent. Predicted Image I pred and a segmentation mask M indicating the foreground pixels. pred The composite function F(I) used to generate

[0078]

number

[0079] can be defined as, where:

[0080]

number

[0081] is the element-wise product, and I e Background pixels in are set to zero. During training, we use the body part semantic segmentation algorithm to segment the ground truth images captured by the witness camera, I, as shown in Figure 9 (Segmentation). gt I is a semantic segmentation of seg To improve the subject segmentation boundaries, pairwise CRFs can be used to refine the predictions of this algorithm. This semantic segmentation is useful for AR / VR rendering.

[0082] Training the neural network to compute F(I) may include training the neural network to optimize a loss function. L=w1L rec +w2L mask +w3L head +w4L temporal +w1L stereo …(1) Here, the weight w i is chosen empirically to ensure that all losses provide similar contributions.

[0083] Instead of using standard l2 or l1 losses in the image domain, the l1 loss can be computed in the feature space of a 16-layer network (e.g., VGG16) trained on an image database (e.g., ImageNet). The loss can be computed as the l-1 distance of the activations from conv1 to conv5 layers. This gives results that are very comparable to using generative adversarial network (GAN) losses, without the overhead of employing a GAN architecture during training. The reconstruction loss L rec can be calculated as follows:

[0084]

number

[0085] Here, M gt =(I seg ≠ background) is a binary segmentation mask that turns off background pixels (see Figure 9), and M pred is the predicted binary segmentation mask, i (·) maps the image to the activations of the conv-i layer of VGG, and ||·|| * is the “saliency reweighted” l1 norm defined later in this section. The predicted binary segmentation mask M predcan be predicted based on a combination of both the visible light information captured for a frame and the IR light captured for a frame. The high contrast between the IR signal of the person and the background with high IR reflectance can be used to distinguish the person from the background and define a segmentation mask. To speed up color convergence, I is weighted to contribute 1 / 10 of the main reconstruction loss. gt and I pred L, defined as the l1 norm between rec An example of the reconstruction loss is shown in Figure 10A.

[0086] Mask Loss L mask The model can then obtain an accurate foreground mask M pred This can be seen as a binary classification task. Foreground pixels have a value y + = 1 and background pixels are assigned y - = 0 is used. The final loss can be defined as: L mask =||M gt -M pred || * …(3) where ||·|| * is the l1 loss with saliency reweighted. Other classification losses, such as logistic loss, can also be considered, but may yield very similar results. An example of a mask loss is shown in Figure 10B.

[0087] Head loss L head can focus the neural network on the head to improve the overall clarity of the face. Similar to the body loss, a 16-layer network (e.g., VGG16) can be used to compute the loss in the feature space. In particular, crop I C I seg We can define a patch for an image I as a patch cropped around its starting pixel, given by its segmentation labels in I and resized to 512 × 512 pixels. The loss can be computed as:

[0088]

number

[0089] An example of head loss is shown in FIG. 10C. time loss L temporal can be used to minimize the amount of flicker between two successive frames. Frame I t and I t-1 The time loss between the t and I t-1 Minimizing the difference between produces a time-blurred result. Hence, the time gradient of the predicted sequence, i.e.

[0090]

number

[0091] Let us define the time gradient of the ground truth sequence, i.e.

[0092]

number

[0093] We can use a loss that tries to match the

[0094]

number

[0095] An example of the calculated time loss is shown in FIG. 10E. Stereo Loss L stereo can be designed for VR and AR applications, where the neural network is applied to the left and right eye views. In this case, the mismatch between the two eyes limits the depth perception and may cause discomfort to the user. Therefore, a loss can be used that ensures self-supervised consistency of the output stereo images. A stereo pair of volumetric reconstructions can be rendered and the image for each eye can be used as input to the neural network, in this case the left image IL matches the ground truth camera viewpoint, and the right image I R is rendered with an offset distance along the x-coordinate (e.g., 65 mm). Then, the right prediction

[0096]

number

[0097] is warped to the left viewpoint using the (known) geometry of the mesh, and the left prediction

[0098]

number

[0099] Compared to Warp Operator I. warp can be defined using a spatial transformer network (STN) using 4-pixel bilinear interpolation and fixed warped coordinates. The loss can be calculated as:

[0100]

number

[0101] An example of stereo loss is shown in FIG. 10D. The calculation of the above losses when training the model can be based on the captured color image data of the scene, the captured wide-field IR image data of the scene, and the depth data based on the captured images of the patterns of IR light on the objects in the scene. Training the model using all these channels improves the performance of the model and the accuracy of the images generated by the model. For example, if two adjacent pixels have different colors and / or albedo values ​​and different depth values, little or no penalty may be applied to the pixels since they are considered to represent different features. However, if two adjacent pixels have very similar colors and albedo values ​​but different depth values, the average of the different depth values ​​may be assigned to both pixels. Thus, taking into account all of the captured color image data of the scene, the captured wide-field IR image data of the scene, and the depth data based on the captured images of the patterns of IR light on the objects in the scene when training and applying the model can enhance its ability to distinguish and render edges and smooth surfaces in the image.

[0102] The above loss receives contributions from all pixels in the image (except for masked pixels). However, imperfections in the segmentation mask may bias the network towards unimportant regions. The pixels with the highest loss may be outliers (e.g., next to the boundary of the segmentation mask). These outlier pixels may dominate the overall loss (see Figure 10F). Therefore, it may be desirable to deweight these outlier pixels to exclude them from the loss, while at the same time deweighting pixels that are easier to reconstruct (e.g., smooth, texture-free regions). To do this, given a residual image x of size W × H × C, set y as the pixel-wise l1 norm along the channels of x and take the minimum and maximum percentile p min and p max can be defined for the value of y. The p component of a pixel in the saliency reweighting matrix of the residual y can be defined as

[0103]

number

[0104] where Γ(i,y) is the i-th percentile from the entire set of y values, and p min , p max , α i is chosen empirically and depends on the task at hand.

[0105] L req and L head This saliency, as a weight for each pixel of the residual y computed with respect to, can be defined as:

[0106]

number

[0107] Where:

[0108]

number

[0109] is the element-wise product. Gamma, defined as the product of a sigmoid and an inverse sigmoid p A continuous formulation of (y) can also be used. No gradients with respect to the reweighting function are computed. Thus, the reweighting function does not need to be continuous for SGD to work. The effect of saliency reweighting is shown in Figure 10F. When saliency reweighting is not used, reconstruction errors occur along object boundaries. Conversely, applying the proposed outlier removal technique forces the network to focus on reconstructing real objects. Finally, as a by-product of saliency reweighting, we can predict cleaner foreground masks when compared to those obtained with semantic segmentation algorithms. The saliency reweighting scheme can be applied only to reconstruction, mask, and head loss.

[0110] 5 illustrates a block diagram of a neural re-rendering module according to at least one exemplary embodiment. The neural re-rendering module 210 can be or include at least one computing device, and can represent virtually any computing device configured to perform the methods described herein. Thus, the neural re-rendering module 210 can include various components that can be utilized to implement the techniques described herein, or different or future versions thereof. As an example, the neural re-rendering module 210 is illustrated as including at least one processor 505 and at least one memory 510 (e.g., a non-transitory computer-readable medium).

[0111] As shown in FIG. 5, the neural re-rendering module includes at least one processor 505 and at least one memory 410. The at least one processor 505 and the at least one memory 510 are communicatively coupled via a bus 515. The at least one processor 505 may be utilized to execute instructions stored in the at least one memory 510, thereby implementing various features and functions described herein, or additional or alternative features and functions. The at least one processor 505 and the at least one memory 510 may be utilized for various other purposes. In particular, the at least one memory 510 may represent one example of various types of memory and associated hardware and software that may be used to implement any one of the modules described herein.

[0112] The at least one memory 510 may be configured to store data and / or information related to the neural re-rendering module 210. For example, the at least one memory 510 may be configured to store the model 420, the plurality of coefficients 425, and the neural network 520. In an example implementation, the at least one memory 510 may be configured to store a code segment that, when executed by the at least one processor 505, causes the at least one processor 505 to select one of the models 420 and / or one or more of the plurality of coefficients 425.

[0113] Neural network 520 may include multiple operations (e.g., convolutions 530-1 through 530-9). The multiple operations, interconnections, and data flows between the multiple operations may be a model selected from model 420. The model shown in neural network (as operations, interconnections, and data flows) is an example implementation. Thus, other models may be used to enhance images as described herein.

[0114] 5, the operations of the neural network 520 include convolutions 530-1, 530-2, 530-3, 530-4, 530-5, 530-6, 530-7, 530-8, and 530-9, convolution 535, and convolutions 540-1, 540-2, 540-3, 540-4, 540-5, 540-6, 540-7, 540-8, and 540-9. Optionally (as shown by the dashed lines), the operations of the neural network 520 can include padding 525, clipping 545, and super-resolution 550. The padding 525 can be configured to pad or add pixels to the input image at the border of the image if the input image needs to be enlarged. Padding may include using pixels adjacent to the border of the image (such as mirror padding). Padding includes adding a number of pixels with values ​​R=0, G=0, B=0 (e.g., zero padding). Clip 545 can be configured to clip any value of R, G, B from 255 and above, and any value from 0 and below. Clip 545 can be configured to clip for other color systems (e.g., YUV) based on the max / min of the color system.

[0115] Super-resolution 550 may include upscaling the resulting image (e.g., ×2, ×4, ×6, etc.) and applying a neural network as a filter to the upscaled image to generate a high-quality image from the relatively low-quality upscaled image. In an example implementation, the filters are selectively applied to each pixel from a number of trained filters.

[0116] In the example implementation shown in FIG. 5, the neural network 520 uses a U-NET-like architecture. This model can implement viewpoint synthesis from 2D images in real time on a GPU architecture. In the example implementation, a fully convolutional model (e.g., no max pooling operator) is used. Additionally, this implementation can use bilinear upsampling and convolution to minimize or eliminate checkerboard artifacts.

[0117] As shown, the neural network 520 architecture includes 18 layers. Nine (9) layers are used for encoding / compressing / reducing / downsampling, and nine (9) layers are used for decoding / decompressing / expanding / upsampling. For example, convolutions 530-1, 530-2, 530-3, 530-4, 530-5, 530-6, 530-7, 530-8, and 530-9 are used for encoding, and convolutions 540-1, 540-2, 540-3, 540-4, 540-5, 540-6, 540-7, 540-8, 540-9 are used for decoding. Convolution 535 may be used as a bottleneck. The bottleneck may be a 1×1 convolution layer configured to reduce the number of input channels of the K×K filter. The neural network 520 architecture may include skip connections between the encoder block and the decoder block. For example, skip connections are shown between convolution 530-1 and convolution 540-9, between convolution 530-3 and convolution 540-7, between convolution 530-5 and convolution 540-5, and between convolution 530-7 and convolution 540-3.

[0118] In an exemplary implementation, the encoder includes N init The convolution starts with convolution 530-1, which consists of a 3 × 3 convolution with a filter, followed by a series of downsampling blocks including convolutions 530-2, 530-3, 530-4, and 530-5. Convolutions 530-2, 530-3, 530-4, 530-5, 530-6, and 530-7 (i ∈ {1, 2, 3, 4}) each have N iTwo convolutional layers with filters can be included. The first layers 530-2, 530-4, and 530-6 can have a filter size of 4×4, stride of 2, and padding of 1, while the second layers 530-3, 530-5, and 530-7 can have a filter size of 3×3 and stride of 1. Thus, each convolution can reduce the size of the input by a factor of two due to strided convolution. Finally, convolutions 530-8 and 530-9 are performed that preserve the two-dimensionality. The output of the convolutions can be passed through a ReLU activation function. In the example implementation, N init =32 and N i =G i N init where G is the increase factor of the filter size after each downsampling block.

[0119] The decoder includes upsampling blocks 540-3, 540-4, 540-5, 540-6, 540-7, 540-8, and 540-9 that mirror but are inverse to the downsampling blocks. Each such block i ∈ {4, 3, 2, 1} consists of two convolutional layers. The first layer 540-3, 540-5, and 540-7 bilinearly upsamples the input and produces N i The second layer 540-4, 540-6, and 540-8 are 2N filters of size 3 × 3 that perform convolution and concatenate the output with the output of its mirrored encoding layer using skip connections. i The filters are used to perform the convolution. The final network output is produced by a final convolution 540-9 with four filters, whose output is passed through a ReLU activation function to produce a reconstructed image and a single channel binary mask of the foreground object. To generate stereo images for VR and AR headsets, the same neural network (with shared weights) is used to enhance both the left and right views. The final output is a refined stereo output pair. Data related to the neural network 520 (e.g., filter size, stride, weights, N init , N i , G i, etc.) may be stored in the model 420 and coefficients 425.

[0120] Returning to Figure 4, the model associated with the architecture of neural network 520 can be trained as described above. The neural network can be trained using the Adam algorithm and weight decay algorithm until it converges (e.g., to the point where the loss does not consistently decrease).

[0121] Random crops of images ranging from 512x512 to 960x896 were used for training. These images can be cropped from the original resolution of the input-output pair. In particular, the random crop has the potential to include head pixels in 75% of the samples, for which head loss is computed. Otherwise, the head loss may be invalid, as the network may not be able to fully recognize the head loss in the input patch. This allows us to obtain high-quality results for faces without ignoring other parts of the body. Using random crops together with a standard l-2 regularization on the network weights may be sufficient to prevent overfitting. With a high-resolution witness camera, the output may be twice the size of the input.

[0122] The percentile range of the saliency reweighting can be empirically set to remove the contributions of imperfect mask boundaries and other outliers without affecting the results. max For =98, p in the range [25,75] min In particular, for the reconstruction loss, p min =50, for head loss p min = 25, and α1 = α2 = 1.1.

[0123] FIG. 6A illustrates a layer of a convolutional neural network without a sparsity constraint. FIG. 6B illustrates a layer of a convolutional neural network with a sparsity constraint. An example implementation of a layered neural network is shown in FIG. 6A as having three layers 605, 610, 615. Each layer 605, 610, 615 can be formed from multiple neurons 620. No sparsity constraint is applied to the implementation shown in FIG. 6A. Thus, every neuron 620 in each layer 605, 610, 615 is network-connected to every neuron 620 in any adjacent layer 605, 610, 615. The neural network shown in FIG. 6A is computationally less complex due to the small number of neurons 620 and layers 605, 610, 615. However, the configuration of the neural network in FIG. 6A may not be easily scaled up to larger network sizes (e.g., connections between neurons / layers) because computational complexity increases as the size of the network increases, and scales nonlinearly with the density of connections.

[0124] When neural networks are scaled up to operate on relatively high dimensional inputs, it can become computationally complex when every neuron 620 in each layer 605, 610, 615 is networked to every neuron 620 in one or more adjacent layers 605, 610, 615. Initial sparsity conditions can be used to lower the computational complexity of neural networks. For example, when a neural network is functioning as an optimization process, limiting the number of neurons and / or connections between layers allows the neural network approach to handle high dimensional data such as images.

[0125] An example of a neural network with a sparsity constraint according to at least one embodiment is shown in Figure 6B. The neural network shown in Figure 6B is configured such that each neuron 620 is connected to only a small number of neurons 620 in adjacent layers 625, 630, 635, thus creating a neural network that is not fully connected and can be expanded to work with higher dimensional data, for example as image enhancement processing. The reduced number of connections compared to a fully networked neural network allows for substantially linear expansion of the number of connections between neurons.

[0126] Alternatively, in some embodiments, neural networks may be used that are fully connected or not fully connected, but in a particular configuration different from that described in connection with FIG. 6B.

[0127] Additionally, some embodiments use convolutional neural networks, which are not fully connected neural networks and therefore less complex than fully connected neural networks. Convolutional neural networks can also utilize pooling or max pooling to reduce the dimensionality (and therefore complexity) of the data flowing through the neural network, thereby reducing the level of computation required.

[0128] FIG. 11 illustrates an example of a computing device 1100 and a mobile computing device 1150 that may be used with the techniques described herein. The computing device 1100 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The computing device 1150 is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smartphones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are for illustrative purposes only and are not intended to limit the implementation of the invention(s) described and / or claimed in this document.

[0129] The computing device 1100 includes a processor 1102, a memory 1104, a storage device 1106, a high-speed interface 1108 connecting to the memory 1104 and a high-speed expansion port 1110, and a low-speed interface 1112 connecting to a low-speed bus 1114 and the storage device 1106. Each of the components 1102, 1104, 1106, 1108, 1110, and 1112 are interconnected using various buses and may be mounted on a common motherboard or in other manners as desired. The processor 1102 may process instructions for execution within the computing device 1100, including instructions stored in the memory 1104 or the storage device 1106, to display graphical information for a GUI on an external input / output device, such as a display 1116 coupled to the high-speed interface 1108. In other implementations, multiple processors and / or multiple buses may be used, as desired, along with multiple memories and multiple types of memories. Also, multiple computing devices 1100 may be connected, each providing a portion of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0130] The memory 1104 stores information within the computing device 1100. In one implementation, the memory 1104 is one or more volatile memory units. In another implementation, the memory 1104 is one or more non-volatile memory units. The memory 1104 may be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0131] The storage device 1106 can provide mass storage for the computing device 1100. In one implementation, the storage device 1106 can be or include a computer-readable medium such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. The computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods such as those described above. The information carrier is a computer or machine-readable medium such as the memory 1104, the storage device 1106, or a memory on the processor 1102.

[0132] The high-speed controller 1108 manages the bandwidth-intensive operations of the computing device 1100, while the low-speed controller 1112 manages the less bandwidth-intensive operations. Such an allocation of functions is merely an example. In one implementation, the high-speed controller 1108 is coupled to the memory 1104, the display 1116 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 1110 that can accept various expansion cards (not shown). In this implementation, the low-speed controller 1112 is coupled to the storage device 1106 and the low-speed expansion port 1114. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled, for example, via a network adapter, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or a router.

[0133] The computing device 1100 may be implemented in many different forms, as shown. For example, the computing device may be implemented as a standard server 1120, or multiple times within a group of such servers. The computing device may also be implemented as part of a rack server system 1124. Additionally, the computing device may be implemented in a personal computer, such as a laptop computer 1122. Alternatively, the components of the computing device 1100 may be combined with other components in a mobile device (not shown), such as device 1150. Each such device may include one or more computing devices 1100, 1150, and the entire system may be made up of multiple computing devices 1100, 1150 that communicate with each other.

[0134] Computing device 1150 includes, among other components, a processor 1152, memory 1164, input / output devices such as a display 1154, a communication interface 1166, and a transceiver 1168. Device 1150 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 1150, 1152, 1164, 1154, 1166, and 1168 are interconnected using various buses, and some of the components may be implemented on a common motherboard or in other manners as appropriate.

[0135] The processor 1152 can execute instructions within the computing device 1150, including instructions stored in the memory 1164. The processor may be implemented as a chipset of chips including separate analog and digital processors. The processor can coordinate other components of the device 1150, such as controlling the user interface, applications run by the device 1150, and wireless communications by the device 1150, for example.

[0136] The processor 1152 can communicate with a user via a control interface 1158 and a display interface 1156 coupled to a display 1154. The display 1154 can be, for example, a TFTLCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other suitable display technology. The display interface 1156 can include appropriate circuitry for driving the display 1154 to present graphical and other information to a user. The control interface 1158 can receive commands from a user and translate them for transmission to the processor 1152. Additionally, an external interface 1162 can be provided in communication with the processor 1152 to enable short-range communication between the device 1150 and other devices. The external interface 1162 can provide, for example, wired communication in some implementations and wireless communication in other implementations, and multiple interfaces can also be used.

[0137] The memory 1164 stores information within the computing device 1150. The memory 1164 may be implemented as one or more of one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 1174 may also be provided and connected to the device 1150 via an expansion interface 1172, which may include, for example, a SIMM (single in-line memory module) card interface. Such expansion memory 1174 may provide additional storage space to the device 1150 or may store applications or other information in the device 1150. In particular, the expansion memory 1174 may include instructions for performing or supplementing the aforementioned processes, and may also include secure information. Thus, for example, the expansion memory 1174 may be provided as a security module of the device 1150 and may be programmed with instructions that enable secure use of the device 1150. Additionally, secure applications may be provided via a SIMM card along with additional information, such as placing identifying information on the SIMM card in an unhackable manner.

[0138] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one implementation, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 1164, expansion memory 1174, or memory on processor 1152, which may be received, for example, via transceiver 1168 or external interface 1162.

[0139] The device 1150 can communicate wirelessly via a communication interface 1166, which may include digital signal processing circuitry as necessary. The communication interface 1166 may provide for communication under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, via a radio frequency transceiver 1168. Additionally, short-range communication may occur, such as using Bluetooth, Wi-Fi, or other such transceivers (not shown). Additionally, a GPS (Global Positioning System) receiver module 1170 may provide additional navigation and location related wireless data to the device 1150, which may be used as appropriate by applications executing on the device 1150.

[0140] Device 1150 can also communicate voice using audio codec 1160, which can receive spoken information from a user and convert it into usable digital information. Audio codec 1160 can similarly generate audible sounds for the user, such as through a speaker in a handset of device 1150. Such sounds can include sounds from a voice call, can include recorded sounds (e.g., voice messages, music files, etc.), and can also include sounds generated by applications running on device 1150.

[0141] The computing device 1150 may be implemented in many different forms, as shown in the figure. For example, the computing device may be implemented as a mobile phone 1180. The computing device may also be implemented as part of a smartphone 1182, personal digital assistant, or other similar mobile device.

[0142] While the above description describes experiencing traditional three-dimensional (3D) content, including accessing a head-mounted display (HMD) device to properly view and interact with such content, the described techniques can also be used for rendering to 2D displays (e.g., left and / or right views displayed on one or more 2D displays), mobile AR, and 3DTV. Furthermore, the use of an HMD device can be cumbersome for a user to continually wear. Thus, a user can utilize an autostereoscopic display to access a user experience with 3D perception without the use of an HMD device (e.g., glasses or headgear). An autostereoscopic display uses optical components to achieve the 3D effect of various images on the same plane and presents such images from multiple viewpoints to create the illusion of a 3D space.

[0143] Autostereoscopic displays can provide images that approximate the three-dimensional (3D) optical properties of real-world physical objects without requiring the use of a head-mounted display (HMD) device. Generally, autostereoscopic displays include a flat panel display, a lenticular lens (e.g., a microlens array), and / or a parallax barrier to redirect images to a number of different viewing regions associated with the display.

[0144] In some instances of autostereoscopic displays, there may be a single location that provides a 3D view of the image content provided by such a display. A user can sit in one location and experience a realistic 3D image with proper parallax and low distortion. As the user moves to another physical location (or changes head position or eye gaze position), the image content may begin to appear less realistic, 2D, and / or distorted. The systems and methods described herein can reconstruct the image content projected from the display to ensure that the user can move around while still experiencing a proper parallax, low distortion rate, and realistic 3D image in real time. Thus, the systems and methods described herein provide the advantage of maintaining and providing 3D image content to the user regardless of user movements that occur while the user is viewing the display.

[0145] 12 illustrates a block diagram of an exemplary output image for providing content on a stereoscopic display, according to at least one exemplary embodiment. In an exemplary implementation, content may be displayed by interleaving a left image 1204A with a right image 1204B to obtain an output image 1205. The autostereoscopic display assembly 1202 illustrated in FIG. 12 represents an assembled display including at least a high-resolution display panel 1207 coupled (e.g., glued) to a lenticular array of lenses 1206. Additionally, the assembly 1202 may include one or more glass spacers 1208 disposed between the lenticular array of lenses and the high-resolution display panel 1207. During operation of the display assembly 1202, the array of lenses 1206 (e.g., a microlens array) and glass spacer 1208 may be designed such that, in a particular viewing condition, a user's left eye sees a first subset of pixels associated with an image, as indicated by viewing ray 1210, while the user's right eye sees a second, mutually exclusive subset of pixels, as indicated by viewing ray 1212.

[0146] A mask may be calculated and generated for each of the left and right eyes. The mask 1200 may be different for each eye. For example, the mask 1200A may be calculated for the left eye, while the mask 1200B may be calculated for the right eye. In some implementations, the mask 1200A may be a shifted version of the mask 1200B. Consistent with embodiments described herein, the autostereoscopic display assembly 1202 may be a glasses-free lenticular three-dimensional display including a plurality of microlenses. In some implementations, the array 1206 may include microlenses in a microlens array. In some implementations, the 3D image may be generated by projecting a portion of a first image (e.g., a first set of pixels) through at least one microlens in a first direction (e.g., to a user's left eye) and projecting a portion of a second image (e.g., a second set of pixels) through at least one other microlens in a second direction (e.g., to a user's right eye). The second image may be similar to the first image, but the second image may be shifted from the first image to simulate parallax, thereby simulating a 3D stereoscopic image for a user viewing the autostereoscopic display assembly 1202.

[0147] FIG. 13 is a schematic flow chart of a process 1300 for implementing the techniques described herein. As shown in FIG. 13, in step 1302, a rendered image based on a Neural Radiance Field (NeRF) volumetric representation of a scene is received. Here, the NeRF representation of the scene is based on captured frames of video data, where each captured frame includes a color image of the scene, a wide-field IR image of the scene captured when the scene is illuminated by a wide-field IR light source, and multiple depth IR images of the scene. Each depth IR image is captured when the scene is illuminated by a different pattern of IR light points, where the illumination by the different patterns of IR light points occurs at different times. The NeRF representation provides a mapping between a position and a viewing direction to a color and optical density at each position in the scene, which allows the scene to be viewed from a new viewpoint, and the NeRF representation provides a mapping between a position and a viewing direction to an IR value for each different pattern of IR light points from the new viewpoint. In step 1304, a synthesis function is defined for generating an enhanced image from the received image. In step 1306, a synthesis function is calculated using the trained neural network based on minimizing a loss function between the predicted images generated by the neural network and the ground truth images captured by the ground truth camera during training.

[0148] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (Application Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be dedicated or general purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device. Various implementations of the systems and techniques described herein may be realized as and / or generally referred to herein as circuits, modules, blocks, or systems that may combine aspects of software and hardware. For example, a module may include functions / operations / computer program instructions executed on a processor (e.g., a processor formed on a silicon substrate, GaAs substrate, etc.) or some other programmable data processing apparatus.

[0149] Some of the above example embodiments are described as a process or method that is depicted as a flowchart. Although the flowchart describes operations as a sequential process, many of the operations can be performed in parallel, simultaneously, or simultaneously. Additionally, the order of operations may be rearranged. A process is terminated when an operation is completed, but may have additional steps not included in the illustration. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0150] The methods discussed above, some of which are illustrated by flowcharts, may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine or computer readable medium, such as a storage medium. A processor may perform the necessary tasks.

[0151] Specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments, however, example embodiments may be embodied in many alternative forms and should not be construed as being limited to only the embodiments set forth herein.

[0152] In this specification, terms such as first, second, etc. may be used to describe various elements, but it can be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another element. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0153] When an element is referred to as being connected or coupled to another element, it will be understood that it can be directly connected or coupled to the other element, or that intervening elements may be present. In contrast, when an element is referred to as being directly connected or coupled to another element, there are no intervening elements present. Other words used to describe the relationship between elements should be interpreted similarly (e.g., directly between elements, directly between adjacent elements, directly adjacent elements, etc.).

[0154] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the exemplary embodiments. As used herein, the singular forms a, an, and the are intended to include the plural forms unless the context clearly dictates otherwise. Furthermore, as used herein, the terms "comprises, comprises, includes, and / or includes" specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0155] It should also be noted that in some alternative implementations, the functions / acts noted may occur out of the order noted in the figures. For example, two figures shown in succession may in fact be executed concurrently or may be executed in the reverse order, depending on the functions / acts involved.

[0156] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which the exemplary embodiments belong. Furthermore, terms, for example, terms defined in commonly used dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly defined in this specification.

[0157] Some of the above exemplary embodiments and corresponding detailed description are presented in terms of software, or algorithms and symbolic representations of operations on data bits within a computer memory. These descriptions and representations are the way those skilled in the art effectively convey the substance of their work to others skilled in the art. An algorithm, as the terminology and as it is used herein, and as it is generally used, is conceived to be a self-consistent sequence of steps leading to a desired result. These steps are steps requiring physical operations of physical quantities. Usually, though not necessarily, these quantities take the form of optical, electrical, or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It is sometimes convenient, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0158] In the above exemplary embodiments, references to symbolic representations of acts and operations (e.g., in the form of a flowchart) that may be implemented as program modules or functional processes include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types to be written and / or implemented using existing hardware with existing structural elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits, field programmable gate array (FPGA) computers, etc.

[0159] It should be borne in mind, however, that these and similar terms are all associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise noted, or as will be apparent from the discussion, terms such as processing or computing, or calculating or determining, such as displaying, refer to operations and processes of a computer system or similar electronic computing apparatus that manipulate and convert data, represented as physical, electronic quantities in the computer system's registers and memory, into other data, also represented as physical quantities in the computer system's memory or registers or other such information storage, transmission, or display devices.

[0160] It should also be noted that the software-implemented aspects of the exemplary embodiments are typically encoded on some form of non-transitory program storage medium or implemented on some type of transmission medium. The program storage medium may be magnetic (e.g., a floppy disk or hard drive) or optical (e.g., a compact disk read-only memory, or CD-ROM), and may be read-only or random access. Similarly, the transmission medium may be twisted wire pairs, coaxial cable, optical fiber, or other suitable transmission medium known in the art. The exemplary embodiments are not limited by these aspects of any given implementation.

[0161] Finally, it should also be noted that, although the appended claims recite particular combinations of features described in this specification, the scope of the present disclosure is not limited to the specific combinations claimed below, but rather extends to encompass any combination of features or embodiments disclosed herein, regardless of whether that particular combination is currently specifically recited in the appended claims.

Claims

1. receiving a rendered image based on a Neural Radiance Field (NeRF) volumetric representation of a scene, the NeRF volumetric representation of the scene based on captured frames of video data, each captured frame including a color image of the scene and a plurality of depth IR images of the scene, each of the depth IR images being captured when the scene is illuminated by a different pattern of IR light points, the NeRF volumetric representation providing a mapping between a position and a viewing direction to a color and optical density at each location in the scene that allows the scene to be viewed from a new viewpoint, the NeRF volumetric representation providing a mapping between a position and a viewing direction to an IR value for each of the different patterns of IR light points from the new viewpoint; defining a combination function for generating an enhanced image from the received image; Calculating the synthesis function using a neural network trained based on a loss function between the predicted image generated by the neural network and a ground truth image captured by a ground truth camera during training; The method includes:

2. Prior to the step of receiving the image, Capturing a 3D model of the scene using a volumetric capture system; rendering the image based on the NeRF volumetric representation; The method of claim 1 further comprising:

3. The method of claim 2 , wherein the ground truth camera and the volume capture system are both pointed at a view during training, and the ground truth camera produces higher quality images than the volume capture system.

4. 4. The method of claim 1, wherein the NeRF volumetric representation of the scene is configured to generate position- and viewpoint-specific color, IR, and density image data of the scene for a plurality of viewing positions and viewing directions relative to the scene.

5. The method of claim 4 , wherein the location-specific and viewpoint-specific IR image data includes wide-field IR image data and patterned IR image data for each of the different patterns of IR light points.

6. 6. The method of claim 1, wherein the loss function comprises a reconstruction loss based on a reconstruction difference between a segmented ground truth image mapped to activations of a layer in a neural network and a segmented predicted image mapped to activations of a layer in a neural network, the segmented ground truth image is segmented by a ground truth segmentation mask to remove background pixels, the segmented predicted image is segmented by a predicted segmentation mask to remove background pixels, and the predicted segmentation mask is predicted based on a combination of both visible light information captured for a frame and IR light captured for a frame.

7. The method of claim 6 , wherein the reconstruction difference is saliency reweighted to downweight the reconstruction difference at pixels above a maximum error or below a minimum error.

8. 8. The method of claim 1, wherein the loss function comprises a mask loss based on a mask difference between a ground truth segmentation mask and a predicted segmentation mask, the predicted segmentation mask being predicted based on a combination of both visible light information captured for a frame and IR light captured for a frame.

9. 9. The method of claim 1, wherein the predicted image is one of a predicted stereo pair of images, and the loss function comprises a stereo loss based on a stereo difference between the predicted stereo pair of images.

10. The method according to any one of claims 1 to 9, wherein the neural network is based on a fully convolutional model.

11. said step of calculating said composite function using a neural network comprising: calculating the composite function at a left eye viewpoint; Calculating the composite function at a right eye viewpoint; The method according to any one of claims 1 to 10, comprising:

12. The method according to any one of claims 1 to 11, wherein the step of calculating the composite function using a neural network is performed in real time.

13. The method of any one of the preceding claims, wherein the colour images of the scene in frames comprise demosaiced colour channels.

14. 13. A method according to any one of claims 1 to 12, wherein the colour image of the scene in a frame comprises three independent non-demosaiced colour channels and the captured frames of video data underlying the NeRF volumetric representation of the scene comprise the three independent non-demosaiced colour channels.

15. a volume capture system configured to render at least one image reconstructed from at least one viewpoint of a captured 3D model, the at least one image being an image rendered based on a Neural Radiance Field (NeRF) volumetric representation of a scene, the at least one image including imperfections, the NeRF volumetric representation providing a mapping between positions and viewing directions to colors and optical densities at each position in the scene that allow the scene to be viewed from a new viewpoint, the NeRF volumetric representation providing a mapping between positions and viewing directions to IR values ​​for each different pattern of IR light points from the new viewpoint; a rendering system configured to receive the at least one image from the volume capture system and generate in real time at least one enhanced image with reduced imperfections of the at least one received image, the rendering system including a neural network configured to generate the at least one enhanced image by training prior to use, the training including minimizing a loss function between predicted images generated by the neural network during training and corresponding ground truth images captured by at least one ground truth camera aligned with the volume capture system during training, the minimization of the loss function being based on captured frames of video data, each captured frame including a color image of a scene and a plurality of depth IR images of the scene, each of the depth IR images being captured when the scene is illuminated by the different patterns of IR light points; A performance capture system comprising:

16. The performance capture system of claim 15 , wherein the at least one ground truth camera is included in the performance capture system during training and is not otherwise included in the performance capture system.

17. 17. The performance capture system of claim 15 or claim 16, wherein the volume capture system includes a single active stereo camera oriented toward a single view and, during training, includes a single ground truth camera oriented toward the single view.

18. 17. The performance capture system of claim 15 or claim 16, wherein the volume capture system includes multiple active stereo cameras oriented at multiple views and, during training, includes multiple ground truth cameras oriented at the multiple views.

19. 17. The performance capture system of claim 15 or claim 16, wherein the performance capture system includes a stereo display configured to display one of the at least one enhanced image as a left eye view and one of the at least one enhanced image as a right eye view.

20. The performance capture system of claim 19 , wherein the performance capture system includes a telepresence communication node.

21. 20. The performance capture system of claim 19, wherein the stereo display is included in an augmented reality (AR) headset.

22. The performance capture system of claim 19 , wherein the stereo display is a head-tracking autostereo display.

Images