Neural re-rendering from 3D models

Abstract

According to one aspect, a method for neural re-rendering includes obtaining a three-dimensional (3D) model representing a physical space scene, wherein the 3D model is constructed from a collection of input images; rendering an image data cache from the 3D model according to a viewpoint, wherein the image data cache represents a reconstructed image from the 3D model; receiving the image data cache by a neural re-rendering network, receiving an appearance code representing an appearance condition by the neural re-rendering network, and transforming the image data by the neural re-rendering network into a re-rendered image having the viewpoint of the image data cache and the appearance condition specified by the appearance code.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation of, and claims priority to, U.S. non-provisional patent application No. 16 / 837,612, filed on April 1, 2020, entitled “NEURAL RERENDERING FROM 3D MODELS,” which claims priority to U.S. Provisional Patent Application No. 62 / 829,478, filed on April 4, 2019, entitled “Neural Rerendering in the Wild,” the disclosure of which is incorporated herein by reference in its entirety.

[0003] This application also claims priority to U.S. Provisional Patent Application No. 62 / 829,478, filed on April 4, 2019, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0004] The description generally relates to neural re-rendering from three-dimensional (3D) models. Background Art

[0005] Recent advances in 3D reconstruction can generate 3D models from a collection of photos, but renderings generated from 3D models may lack the realism and diversity of real-world images. In addition, some traditional techniques restore a single appearance of the scene or restore an average appearance, which causes the rendering to look inconsistent or blurry, and some traditional techniques may hallucinate the positions of transient objects. In addition, generating images from 3D models in outdoor scenes is particularly challenging because the outside world looks very different in different weather or at different times of the day, and it is relatively difficult to characterize these changes using a model (e.g., a single model). Summary of the invention

[0006] According to one aspect, a method for neural re-rendering includes obtaining a three-dimensional (3D) model representing a physical space scene, wherein the 3D model is constructed from a collection of input images; rendering an image data cache from the 3D model according to a viewpoint, wherein the image data cache represents an image reconstructed from the 3D model; receiving the image data cache by a neural re-rendering network; receiving an appearance code representing an appearance condition by the neural re-rendering network; and transforming the image data cache by the neural re-rendering network into a re-rendered image having a viewpoint of the image data cache and an appearance condition specified by the appearance code. Non-transitory computer-readable media and neural re-rendering systems are also provided for these operations.

[0007] The method (or non-transitory computer-readable medium or neural re-rendering system) may include any of the following features (or any combination thereof). In some examples, the appearance condition of the re-rendered image is different from the appearance condition of the input image having the same viewpoint as the viewpoint of the re-rendered image. In some examples, the appearance condition of the re-rendered image is the same as the appearance condition of the input image having the same viewpoint as the viewpoint of the re-rendered image. The method may include receiving, by a neural re-rendering network, a semantic tag map indicating a location of a transient object in a scene, wherein the transformation includes generating the re-rendered image using the semantic tag map as a guide so that the transient object is deleted from the re-rendered image. The image data cache may include the semantic tag map. The method may include receiving, by an image-to-image conversion network, an image data cache, and generating, by the image-to-image conversion network, a semantic tag map based on one or more portions of the image data cache. The image data cache includes multiple layers. The multiple layers include a color layer and a depth layer. The transformation includes generating the re-rendered image based on the color layer and the depth layer. The method may include generating a calibration dataset using the 3D model, the calibration dataset including a plurality of image pairs, wherein each pair includes an input image and an image data cache rendered from the 3D model, the input image and the image data cache being from the same viewpoint, and the image data cache being pixel-calibrated with the input image, generating one or more semantic label maps based on portions of the calibration dataset, wherein the semantic label maps indicate locations of transient objects and non-transient objects, and training a neural re-rendering network using the calibration dataset and the semantic label maps. The method may include training an appearance encoder using the calibration dataset with a triplet loss to learn an appearance code for a scene.

[0008] According to one aspect, a non-transitory computer readable medium storing executable instructions, when executed by at least one processor, is configured to cause the at least one processor to obtain a three-dimensional (3D) model representing a physical space scene, wherein the 3D model is constructed from a collection of input images, render an image data cache from the 3D model according to a viewpoint, wherein the image data cache represents an image reconstructed from the 3D model, receive the image data cache through a neural re-rendering network, receive an appearance code representing an appearance condition by the neural re-rendering network, and transform the image data cache into a re-rendered image by the neural re-rendering network, so that the re-rendered image has the appearance condition specified by the appearance code and the viewpoint of the image data cache. Systems and methods are also provided with these features.

[0009] According to some aspects, a non-transitory computer-readable medium (or system or method) may include one or more of the above / below features (or any combination thereof). The executable instructions include causing at least one processor to receive a semantic label map indicating a location of a transient object in a scene through a neural re-rendering network, and generating a re-rendered image using the semantic label map as a guide by the neural re-rendering network, so that the transient object is deleted from the re-rendered image. The image data cache includes a semantic label map. The executable instructions include instructions for causing at least one processor to receive an image data cache through an image-to-image conversion network and for the image-to-image conversion network to generate a semantic label map based on the image data cache. The executable instructions include instructions for causing at least one processor to receive a ground truth image associated with the image data cache through a semantic segmentation network, and for the semantic segmentation network to generate a semantic label map based on the ground truth image. The image data cache includes multiple layers, wherein the multiple layers include a color layer and a depth layer, and the instructions to be transformed include instructions for generating a re-rendered image based on the color layer and the depth layer.

[0010] According to one aspect, a neural re-rendering system includes a 3D (3D) generator configured to generate a 3D model representing a physical space scene based on a collection of input images; a neural re-rendering network configured to transform a cache of image data rendered from the 3D model to a re-rendered image; an appearance encoder configured to encode multiple appearance conditions of the scene into appearance codes, wherein the neural re-rendering network is configured to receive the appearance codes so that modification of the appearance codes causes a rendered image to be generated according to different appearance conditions; and a neural network trainer configured to train the neural re-rendering network and the appearance encoder using training data. Methods and non-transitory computer-readable media are also provided with these features.

[0011] According to some aspects, a neural re-rendering system (or method or non-transitory computer-readable medium) may include one or more of the above / below features (or any combination thereof). The neural network trainer is configured into a neural re-rendering network and an appearance encoder according to multiple training stages, wherein the multiple training stages include an appearance training stage in which the appearance encoder is trained, a re-rendering training stage in which the neural re-rendering network is trained, and an adjustment training stage in which the appearance encoder and the neural re-rendering network are jointly trained. The 3D model is a point cloud generated by structure from motion (Sfm) and multi-view stereo (MVS) techniques. The image data cache includes multiple layers, wherein the multiple layers include a color layer, a depth layer, and a semantic tag mapping layer indicating the location of transient objects. The neural re-rendering network is a first neural network, and the appearance encoder is a second neural network.

[0012] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Illustration of a neural re-rendering system according to an aspect.

[0014] Figure 2 A 3D model generator configured to generate a 3D model according to an aspect is illustrated.

[0015] Figure 3 Illustrated is an example of image data caching according to an aspect.

[0016] Figure 4 Illustrated is a transient object detector having a semantic segmentation network configured to generate a semantic label map according to an aspect.

[0017] Figure 5 Illustrated is a transient object detector having an image-to-image translation network configured to generate a semantic label map according to an aspect.

[0018] Figure 6 Illustrated is a neural network trainer configured to train an appearance encoder and a neural re-rendering network using a calibration dataset according to an aspect.

[0019] Figure 7 Illustrated is a calibration data set generator configured to generate a calibration data set to be used as training data according to an aspect.

[0020] Figure 8 Illustrated are examples of a 3D model and a calibration dataset according to an aspect.

[0021] Fig. 9 Illustrated is an example of a training architecture for a neural re-rendering system according to an aspect.

[0022] Fig.10 Illustrated is an example of a neural network trainer configured to perform multiple training phases according to an aspect.

[0023] Fig.11 Illustrated is an example of a training architecture for a neural re-rendering system according to an appearance training phase, according to one aspect.

[0024] Fig.12 Illustrated is an example of computing a triplet loss for training an appearance encoder according to an aspect.

[0025] Fig.13 Illustrated is an example of calculating a triplet loss for training an appearance encoder according to an aspect.

[0026] Fig.14 Illustrated is an example of a training architecture for a neural re-rendering system according to a re-rendering training phase according to one aspect.

[0027] Fig.15 An example of generating a semantic tag map according to an aspect is illustrated.

[0028] Fig.16 An example of generating a semantic tag map according to another aspect is illustrated.

[0029] Fig.17 FIG. 1 illustrates a training architecture for a neural re-rendering system according to an aspect based on an adjusted training phase.

[0030] Fig.18 Illustrated is an example of a neural re-rendering system according to an aspect.

[0031] Fig.19 The diagram depicts a flow diagram of example operation of a neural re-rendering system according to an aspect.

[0032] Fig. 20 An example computing device illustrating a neural re-rendering system according to an aspect is illustrated. DETAILED DESCRIPTION

[0033] The present disclosure relates to a neural re-rendering system that records, models, and re-renders scenes under changing appearance conditions such as seasons and time of day, which in some examples refers to total scene capture. The neural re-rendering system can solve one or more of the above difficulties. The neural re-rendering system can obtain a collection of photos (e.g., Internet photos) of a scene (e.g., an outdoor scene, a tourist landmark) and apply three-dimensional (3D) reconstruction to register the photos and approximate the scene as a 3D model (e.g., a point cloud, a texture mesh). For each photo, the neural re-rendering system renders the scene points (if a point cloud is used) into an image data cache (e.g., a frame buffer, a depth buffer, a G buffer), and then uses a neural re-rendering network (e.g., a neural network) to re-render the image data cache into a more realistic image. The neural re-rendering network can be considered an image conversion network that transforms the image data cache into a re-rendered image, where the re-rendered image is a relatively realistic image with high detail and resolution. The neural re-rendering system can train the neural re-rendering network to learn the mapping of these initial renderings (e.g., image data caches) to actual photos (e.g., original input images). The model is evaluated on multiple publicly available image datasets spanning a wide range of lighting conditions.

[0034] The neural re-rendering network also receives an appearance code (e.g., a latent appearance vector) and a semantic tag map (e.g., a semantic mask) indicating the location of a transient object such as a pedestrian, wherein the neural re-rendering network can generate multiple outputs of images with different appearance conditions using the image data cache as input (e.g., a single input). In some examples, the appearance condition need not be the same as a real photo taken from the same vantage point (with a fixed appearance). For example, the appearance condition of the re-rendered image can be different from the appearance condition of the input image (e.g., the corresponding original image) having the same viewpoint as the viewpoint of the re-rendered image. For example, the original image can be sunny, but the re-rendered image can be cloudy (or any appearance condition specified by the appearance code). In some examples, the appearance code is an interpolation of the appearance codes of the two input images. Note that the re-rendered image can be generated according to any type of appearance condition. In addition, the adjustment of the neural re-rendering network on the semantic tag map can reduce or eliminate the illusion of the location of transient objects and provide the ability to remove transient objects (e.g., people) from the re-rendered image.

[0035] Figures 1 to 18 Various aspects of a neural re-rendering system 100 are illustrated. Figure 1 An example of a neural re-rendering system 100 according to one aspect is illustrated. The neural re-rendering system 100 can be trained to render images of a scene (e.g., an outdoor scene) with relatively high realism under multiple (or all) possible lighting and weather conditions. In some examples, the neural re-rendering system 100 is used in virtual reality applications and / or augmented reality applications. In some examples, the neural re-rendering system 100 can be used in video game applications. In some examples, the neural re-rendering system 100 can be used for an image mapping service (e.g., a web mapping service). Using a photo collection 101 of input images 102 of a particular scene, the neural re-rendering system 100 can capture the appearance space (e.g., the complete appearance space) of the scene by modeling the scene under multiple appearance conditions in which the scene can be observed. The appearance conditions can include lighting conditions, weather conditions, and transient states (e.g., rain, snow, sunrise, cloudy, crowded, etc.). The neural re-rendering system 100 can then render an image from any arbitrary viewpoint of the scene under any appearance condition. In other words, the neural re-rendering system 100 may receive the photo collection 101 of input images 102 and generate photo-realistic novel viewpoints under multiple appearance conditions represented by the appearance codes 142 .

[0036] Due to the sheer diversity of appearance, for example, scenes can look dramatically different under nighttime lighting, during special events, or in extreme weather, the techniques discussed herein can overcome several technical challenges, either individually or in combination. In addition, the techniques discussed herein can overcome several technical challenges, either individually or in combination, for rendering novel viewpoints that are photo-realistic under multiple appearance conditions using publicly available photographs (e.g., Internet photos) as input images 102. In some examples, the input images 102 of the photo collection 101 include images from different cameras, images from different times of the day, images with different lighting conditions, images including transient objects (e.g., human occlusions, etc.), and / or images with imperfect calibration. In some examples, the input images 102 include a collection of uncontrolled Internet images (e.g., as opposed to carefully captured continuous footage). In some examples, the input images 102 are a collection of community photos of a particular scene. To this end, the various techniques discussed herein can be applied, either individually or in combination.

[0037] The neural re-rendering system 100 can generate or construct a three-dimensional (3D) model 104 using an input image 102. The 3D model 104 can be an explicit geometric approximation of a scene. In some examples, the 3D model 104 is considered to be an explicit intermediate 3D representation that is used as a "scaffolding" to predict an image. Figure 2 The details of the 3D model 104 are further explained. The neural re-rendering system 100 can then render an image data cache 106 from the 3D model 104. The image data cache 106 can be rendered from the 3D model 104 according to any viewpoint (including new viewpoints not captured by the input image 102). The neural re-rendering system 100 can render the 3D model 104 into the image data cache 106 according to a first viewpoint or render the 3D model 104 into the image data cache 106 according to a second viewpoint, where the second viewpoint is different from the first viewpoint. In addition, the first viewpoint and / or the second viewpoint can be new (e.g., not represented in the input image 102). In some examples, the image data cache 106 can be considered as a factorized representation of the input image 102, which separates viewpoints, appearance conditions, and transient objects. A realistic image (e.g., a re-rendered image 110) is rendered from these factorized representations.

[0038] In some examples, image data cache 106 is one or a series of caches that store information about scene geometry at the pixel level (e.g., at each pixel). In some examples, image data cache 106 is a physical memory storage area that stores lighting related data. In some examples, image data cache 106 is a 3D reconstructed image (e.g., rendered from 3D model 104). In some examples, image data cache 106 is considered a frame buffer (or deep frame buffer). In some examples, image data cache 106 is considered a rough initial image. In some examples, image data cache 106 is a deferred shading buffer. In some examples, image data cache 106 is a geometry cache (G-buffer). Reference Figure 3 The details of the image data buffer 106 are further explained.

[0039] The neural re-rendering system 100 then uses the neural re-rendering network 108 to generate a re-rendered image 110 from the image data cache 106. The neural re-rendering network 108 can transform the approximate initial scene rendering (e.g., the image data cache 106) into a realistic image (e.g., the re-rendered image 110). In other words, the neural re-rendering network 108 can generate a realistic image (e.g., the re-rendered image 110) from the initial rough image (e.g., the image data cache 106). The re-rendered image 110 can be a realistic re-rendering of the image data cache 106.

[0040] The input to the neural re-rendering network 108 is the image data cache 106 (e.g., a G-buffer, a deferred shading frame buffer, where each rendered pixel stores reflectivity (color), depth, and other attributes), and the output of the neural re-rendering network 108 is a realistic view under different appearance conditions (e.g., a re-rendered image 110). As discussed later in this disclosure, the neural re-rendering network 108 is trained to learn the transformation from the initial renderer (e.g., the image data cache 106) to a photo-realistic image (e.g., the re-rendered image 110) using training data (which is used by the neural re-rendering system 100 as an align input-output pair). In some examples, the re-rendered image 110 can have a higher quality (e.g., resolution, detail) than the quality of the image data cache 106.

[0041] The neural re-rendering network 108 may receive an appearance code 142 (e.g., an appearance vector having values) that specifies one or more appearance conditions to be included in the re-rendered image 110. In some examples, the neural re-rendering network 108 receives the appearance code 142 at a bottleneck layer of the neural re-rendering network 108. As explained later in this disclosure, an appearance encoder 138 (e.g., a neural network) is trained to encode any appearance condition of a scene, which generates the appearance code 142. In some examples, the appearance code 142 is learned from the input image 102 during training of the neural re-rendering system 100. The appearance code 142 may summarize or represent weather and lighting conditions and any factors that may affect the appearance of the scene. In other words, the value of the appearance code 142 may determine the appearance condition of the re-rendered image 110, and adjustment of the value of the appearance code 142 may change one or more appearance conditions of the scene.

[0042] If the appearance code 142 indicates one or more first appearance conditions (e.g., snowy, cloudy), the neural re-rendering network 108 transforms the image data cache 106 into a re-rendered image 110 such that the re-rendered image 110 includes the one or more first appearance conditions (e.g., snowy, cloudy). Then, the appearance code 142 may be modified to indicate one or more second appearance conditions (e.g., sunny), and the neural re-rendering network 108 transforms the image data cache 106 into a re-rendered image 110 such that the re-rendered image 110 includes the one or more second appearance conditions (e.g., sunny). Since the appearance code 142 is trained to capture multiple appearance conditions of the scene (e.g., any or all appearance conditions), the re-rendered image 110 may be generated according to any appearance condition (or any viewpoint) by adjusting the value of the appearance code 142. The appearance modeling discussed herein may avoid (or reduce) the generation of re-renderings having a single (or average) appearance of the scene, thereby making the re-rendered image 110 more realistic and consistent. Further, by using the appearance code 142 as an input to the neural re-rendering network 108 (at the bottleneck layer), the neural re-rendering network 108 can generate different outputs for a given input (e.g., a scene from a certain viewpoint under multiple different appearance conditions).

[0043] In some examples, the semantic tag map 124 is used as another input to the neural re-rendering network 108 to allow the neural re-rendering network 108 to predict the location of the transient object 125. For example, the semantic tag map 124 can identify information of one or more categories (e.g., on a pixel-by-pixel basis). In some examples, the semantic tag map 124 indicates multiple transient objects 125 of different categories (e.g., a car and two pedestrians). The semantic tag map 124 can include data identifying the location of one or more transient objects 125. In order to prevent (or reduce) the neural re-rendering network 108 from hallucinating the location of the transient object 125 on the original photo (e.g., the input image 102), the neural re-rendering system 100 is constructed so that the semantic tag map 124 is used as another input to the neural re-rendering network 108. In some examples, the semantic tag map 124 is derived from the input image 102. In some examples, the semantic tag map 124 is derived from a portion of the image data cache 106. In some examples, the semantic tag map 124 is included as part of the image data cache 106.

[0044] The neural re-rendering network 108 is a neural network (e.g., a deep neural network). In some examples, the neural re-rendering network 108 is an image conversion network that defines a neural network, wherein the image conversion network is modified to receive the appearance code 142 at a bottleneck layer of the neural re-rendering network 108 and receive the semantic tag map 124 as another input to the neural re-rendering network 108. In some examples, the appearance code 142 (received by the neural re-rendering network 108) corresponds to the appearance code 142 of the original photo (e.g., the input image 102) having the same viewpoint. For example, if the original photo is sunny, the re-rendered image 110 is sunny. In some examples, the appearance code 142 (received by the neural re-rendering network 108) corresponds to an appearance condition that is different from the appearance condition of the input image 102 having the same viewpoint as the re-rendered image 110. For example, the original photo may be sunny, but the appearance code 142 (received by the neural re-rendering network 108) specifies the appearance condition as cloudy.

[0045] The neural re-rendering network 108 can be an interconnected group of nodes, each node representing an artificial neuron. The nodes are interconnected in layers, and the output of one layer becomes the input of the next layer. The neural network transforms the input received by the input layer (e.g., the image data cache 106), transforms it through a series of hidden layers, and produces an output (e.g., the re-rendered image 110) via the output layer. Each layer consists of a subset of the node set. The nodes in the hidden layer are fully connected to all nodes in the previous layer and provide their output to all nodes in the next layer. The nodes in a single layer operate independently of each other (i.e., do not share connections). The nodes in the output provide the transformed input to the request process.

[0046] In some examples, the neural network is a convolutional neural network, which is a neural network that is not fully connected. Therefore, the complexity of a convolutional neural network is lower than that of a fully connected neural network. In addition, as with a 2D convolution operation, the same set of learned weight parameters can be applied to node neighborhoods in a 2D grid of network layers. The convolutional neural network can also utilize pooling or max pooling to reduce the dimensionality (and therefore complexity) of the data flowing through the neural network, and therefore it can reduce the required level of computation. This makes the output computation in a convolutional neural network faster than in a neural network. In some examples, the neural re-rendering network 108 includes a symmetric encoder-decoder with skip connections. In some examples, the neural re-rendering network 108 includes a multi-scale patch GAN discriminator with three scales and employs an LSGAN loss. As a reconstruction loss, the neural re-rendering network 108 uses conv i∈[1,5] of VGG. i，2 The perceived loss of the assessment.

[0047] Figure 2 The 3D model generator 112 is illustrated to generate a 3D model 104 using an input image 102. In some examples, the 3D model generator generates the 3D model 104 based on structure from motion (SfM) 103 and / or multi-view stereo (MVS) 105. SfM 103 can be a photogrammetric range imaging technique for estimating 3D structure from a 2D image sequence that can be coupled with a local motion signal. In some examples, SfM 103 is used to compose images, for example, to estimate the photo position, its orientation, and camera parameters. In some examples, MVS 105 obtains position and orientation information from SfM 103 and creates a 3D model 104 (e.g., a 3D dense point cloud). In some examples, the 3D model 104 is a point cloud 104a (e.g., a 3D dense point cloud). The point cloud 104a can be a set of data points in space that represent the 3D structure of a scene. These points represent the X, Y, and Z geometric coordinates of a single point on the underlying sampled surface. In some examples, the 3D model 104 is a text mesh 104b. The text mesh 104b is a 3D mesh comprising polygons that use reference points in the X, Y, and Z axes to define a shape having height, width, and depth.

[0048] Figure 3An example of an image data cache 106 is illustrated. The image data cache 106 may include multiple layers 114. Each layer 114 may include different categories of image data about a particular viewpoint. The layers 114 may include color (e.g., red, green, blue (RGB)) 116, depth 118, normals 120, semantic tag maps 124, and / or other image-related data 126. The color 116 (or color layer) may be an RGB diffuse color vector, also known as reflectivity. The depth 118 (or depth layer) is a depth map with object depth values. The normal 120 (or normal layer) is a 3D normal vector used to determine the orientation of a surface (e.g., the gradient of its depth in the 2D domain of the image data cache 106). Other image-related data 126 may include a specular intensity float, a light source position and color vector, and a position vector of the camera.

[0049] In some examples, the semantic tag map 124 may include data identifying the location of one or more transient objects 125. However, the semantic tag map 124 may be used to identify the type (or classification) of any type of data (e.g., ground, sky, building, etc.). Although the description discussed herein refers to the semantic tag map 124 as identifying transient objects 125 (or different categories of transient objects 125), the semantic tag map 124 may classify other types of information, where the neural re-rendering network 108 can be instructed to delete any type of object (in addition to or in addition to transient objects 125). The transient object 125 may be image data or image fragments that are determined to be likely to move (e.g., people, cars, objects that are not normally in the scene, etc.).

[0050] As explained later in this disclosure, in some examples, the semantic tag map 124 can be used to generate a re-rendered image 110 that does not include a transient object 125 or certain categories of transient objects 125 such as people. In addition, adjusting the neural re-rendering network 108 on the semantic tag map 124 during training can help reduce the illusion of the location of the transient object 125. In general, the semantic tag map 124 can indicate the segmentation of the digital image into multiple segments, which are used to locate objects and boundaries. In some examples, the semantic tag map 124 can specify labels for pixels (e.g., each label corresponds to a different category), wherein the generation of the semantic tag map 124 can involve assigning labels to each pixel in the image so that pixels with the same label share certain characteristics. In some examples, the semantic tag map 124 is a semantic mask indicating the location of the transient object 125. In some examples, the semantic tag map 124 is a segmentation or semantic segmentation that identifies the location of the transient object 125. In some examples, the semantic tag map 124 is a segmentation map that identifies the location of the transient object 125. In some examples, the semantic label map 124 is included as part of the image data cache 106. In some examples, the semantic label map 124 is separate from the image data cache 106. In some examples, the semantic label map 124 is separated from the image data cache 106 using a semantic segmentation network 130 (see reference Figure 4 102a). In some examples, an image-to-image conversion network 132 (such as the one described in reference 102) is used to generate a semantic label map 124 from an input image 102 (or a ground truth image 102a). Figure 5 As discussed, a semantic tag map 124 is generated from the image data cache 106.

[0051] Figure 4 An example of a neural re-rendering system 100 including a transient object detector 128 is illustrated according to one aspect. In some examples, during training of the neural re-rendering network 108, the transient object detector 128 is configured to generate a semantic label map 124 from a corresponding input image 102 (or ground truth image 102a). In some examples, the transient object detector 128 includes a semantic segmentation network 130 (e.g., a neural network). During training of the neural re-rendering network 108, the semantic segmentation network 130 is configured to receive the input image 102 (or ground truth image 102a) and generate the semantic label map 124. In some examples, the semantic label map 124 is included as part of an image data cache 106 corresponding to the input image 102. Then, when generating a re-rendered image 110 from the image data cache 106, the semantic label map 124 can be used to remove transient objects 125.

[0052] Figure 5An example of a neural re-rendering system 100 with a transient object detector 128 according to another aspect is illustrated. In some examples, after training the neural re-rendering network 108 (e.g., during testing or operation), a new viewpoint may be generated, wherein (as discussed above) an image data cache 106 with the novel viewpoint is generated from the 3D model 104. However, at this point, the image data cache 106 may not include a semantic label map 124. In some examples, the transient object detector 128 is configured to generate the semantic label map 124 from a portion of the image data cache 106. In some examples, the transient object detector 128 uses the color layer 116 and the depth 118 of the image data cache 106 to generate the semantic label map 124. The transient object detector 128 includes an image-to-image conversion network layer 132 (e.g., a neural network). The image-to-image conversion network 132 is configured to receive the color 116 and the depth 118 of the image data cache 106 and generate the semantic label map 124. In some examples, the semantic markup map 124 is then included as part of the image data cache 106. Then, when a re-rendered image 110 is generated from the image data cache 106, the semantic markup map 124 can be used to delete the transient objects 125.

[0053] Figure 6 An example of a neural re-rendering system 100 with a neural network trainer 148 is illustrated according to one aspect. The neural network trainer 148 is configured to train the network using the restored viewpoint parameters of each input image 102 by generating a calibration dataset 146 to render an image data cache 106 of a scene from the same view, e.g., calibrated at a pixel level. The calibration dataset 146 is considered training data for the neural re-rendering system 100. As further explained later in this disclosure, the calibration dataset 146 includes a plurality of image pairs, where each pair includes an input image 102 and an image data cache 106, which is rendered from a 3D model 104. For a particular pair, the input image 102 and the image data cache 106 are from the same viewpoint, and the input image 102 is pixel-calibrated with the image data cache 106. The neural network trainer 148 is configured to train the appearance encoder 138 and the neural re-rendering network 108 using the calibration dataset 146, where the calibration dataset 146 is training data for the appearance encoder 138 and the neural re-rendering network 108.

[0054] Figure 7 An example of a neural re-rendering system 100 is illustrated having a calibration dataset generator 160 configured to generate a calibration dataset 146 using a 3D model 104 . Figure 8 1 and 2 are perspective views of the 3D model 104 and the calibration dataset 146. The calibration dataset 146 includes the 3D model 104 and the calibration dataset 146 obtained by deriving the calibration dataset from each input image 102 (Ii )'s viewpoint v i The rendered image and the real image of the rendered 3D model 104 are i The calibration data set generator 160 may generate an image data buffer 106 (B i ), which may include color 116 (e.g., reflectivity), normals 120, depth 118, and any other image-related data 126. In some examples, image data cache 106 uses only color 116 and depth 118, and renders 3D model 104 by splatting using a z-buffer with a one-pixel radius.

[0055] refer to Figure 7 and 8 , the calibration dataset 146 includes a first set with input images 102 and a second set with image data cache 106. The input images 102 are used to construct the 3D model 104. For each input image 102 in the photo collection 101, the calibration dataset generator 160 renders the 3D model 104 from the estimated viewpoint of the image to obtain the calibration dataset 146.

[0056] exist Figure 7-Figure 8In the example of , input images 102 include image 102-1, image 102-2, and image 103-3, and image data cache 106 includes image data cache 106-1, image data cache 106-2, and image data cache 106-3. Image 102-1 and image data cache 106-1 are considered a first pair, where image data cache 106-1 is from the same viewpoint as image 102-1, and image data cache 106-1 is pixel-calibrated with image 102-1. For example, calibration dataset generator 160 is configured to receive image 102-1 (with a particular viewpoint) and generate image data cache 106-1 from 3D model 104 with the same viewpoint as image 102-1. Image 102-2 and image data cache 106-2 are considered a second pair, where image data cache 106-2 is from the same viewpoint as image 102-2, and image data cache 106-2 is pixel-calibrated with image 102-2. For example, the calibration data set generator 160 is configured to receive the image 102-2 (having a particular viewpoint) and generate the image data cache 106-2 from the 3D model 104 having the same viewpoint as the image 102-2. The image 102-3 and the image data cache 106-3 are considered a third pair, where the image data cache 106-3 is from the same viewpoint as the image 102-3, and the image data cache 106-3 is pixel-calibrated with the image 102-3. For example, the calibration data set generator 160 is configured to receive the image 102-3 (having a particular viewpoint) and generate the image data cache 106-3 from the 3D model 104 having the same viewpoint as the image 102-3.

[0057] Fig. 9 An example of an overall training architecture according to one aspect is illustrated. The neural network trainer 148 is configured to train the appearance encoder 138 (using the triplet loss 164) and the neural re-rendering network 108 (using the loss 166) based on the calibration dataset 146, which includes pairs of input images 102 and image data caches 106. The pairs may include a first pair having an input image 102-1 and an image data cache 106-1. The first pair is used as an example, where the number of pairs may be hundreds or thousands.

[0058] According to one aspect of neural network training, the neural network trainer 148 provides the image data cache 106-1 to the neural re-rendering network 108 and transforms the image data cache 106-1 into a re-rendered image 110-1 from the same viewpoint as the image data cache 106-1. In some examples, the neural re-rendering network 108 receives the color 116 and depth 118 of the image data cache 106-1 and transforms the color 116 and depth 118 into the re-rendered image 110-1. In some examples, the neural re-rendering network 108 is configured to perform an image-to-image conversion. The neural network trainer 148 is then configured to calculate a loss 166 using the re-rendered image 110-1 and the input image 102-1. In some examples, the loss 166 includes a perceptual loss and / or an adversarial loss. In some examples, the loss 166 includes a generative adversarial network (GAN) loss and a VGG (perceptual) loss. In some examples, the input image 102-1 is an original photo, and the original photo is compared to the photo generated by the neural re-rendering network 108, and the difference between the two can be reflected by the loss 166. In other words, the image data cache 106-1 is fed to the neural re-rendering network 108 and the input image 102-1 is used to train the neural re-rendering network 108 (e.g., to teach the neural re-rendering network 108 that the re-rendered image 110-1 should look like the input image 102-1).

[0059] As indicated above, the same viewpoint can appear in different weather conditions, different lighting, and / or different times of day, etc. In some examples, if the neural re-rendering system 100 does not include an appearance encoder 138, the neural re-rendering network 108 can learn an average appearance of the scene and / or the appearance of each viewpoint in the training data. However, to overcome this technical challenge, the neural re-rendering system 100 implements neural re-rendering as a multi-model image synthesis task, where the neural re-rendering network 108 is conditioned on a desired appearance.

[0060] For example, the neural network trainer 148 trains the appearance encoder 138 to learn to encode the appearance of the input image 102 into the appearance code 142, where the appearance code 142 is provided as a separate input (e.g., at a bottleneck layer) into the neural re-rendering network 108. In addition, the appearance encoder 138 is trained using a triplet loss 164, which will be further explained later in this disclosure. The appearance encoder 138 can be a neural network. In some examples, the number of layers of the appearance encoder 138 is less than the number of layers included in the neural re-rendering network 108 (e.g., the appearance encoder 138 is a neural network that is smaller than the neural re-rendering network 108).

[0061] The neural network trainer 148 can train the appearance code 142 so that images with similar appearance have similar codes. In some examples, the appearance code 142 includes a latent appearance code (or a latent appearance vector that defines values ​​or coefficients). The appearance encoder 138 receives as input the input image 102 and the image data buffer 106 and computes the appearance code 142, which captures the input image 102 (e.g., the output domain I i ), which may be obtained from the image data cache 106 (e.g., input domain B i ) is derived. By having the appearance encoder 138 observe the image data cache 106, the appearance encoder 138 can learn a relatively complex appearance model by correlating the lighting in the input image 102 with the scene geometry in the image data cache 106. The appearance code 142 can summarize or represent weather and lighting conditions and any factors that may affect the appearance of the scene. The appearance code 142 generally includes information about the appearance, such as the time of day, whether it is sunny or cloudy, etc. The neural re-rendering network 108 then generates a re-rendered image 110 conditioned on the viewpoint and the appearance code 142 corresponding to the image data cache 106.

[0062] Fig.10 An example of a neural network trainer 148 configured to perform a staged training operation is illustrated according to one aspect. In some examples, the neural network trainer 148 trains the appearance encoder 138 and the neural re-rendering network 108 using the calibration dataset 146 according to a plurality of training stages 150. In some examples, the training stages 150 are performed sequentially by the neural network trainer 148.

[0063] In some examples, the training phase 150 includes an appearance training phase 152 that trains the appearance encoder 138 to estimate an embedding space of appearance conditions of a scene, a re-rendering training phase 154 that trains the neural re-rendering network 108 to transfer an initial rendering (e.g., the image data cache 106) to a realistic image (e.g., the re-rendered image 110), and an adjustment training phase 158 that (jointly) trains the appearance encoder 138 and the neural re-rendering network 108. In some examples, instead of jointly training the appearance encoder 138 and the neural re-rendering network 108, the neural network trainer 148 can perform a staged training process that initially trains the appearance encoder 138 during the appearance training phase 152 using a proxy style-based loss technique (which captures the style of the image).

[0064] Then, during the re-rendering training phase 154, the neural network trainer 148 trains the neural re-rendering network 108 using the fixed appearance embeddings from the already trained appearance encoder 138. In some examples, during the re-rendering training phase 154, the neural network trainer 148 performs semantic conditioning 156 on the neural re-rendering network 108 so that the neural re-rendering network 108 can learn to ignore transient objects 125 rather than trying to hallucinate their locations. The semantic conditioning 156 can refer to conditioning the neural re-rendering network 108 on the semantic label map 124. For example, transient objects 125 (e.g., pedestrians and cars) are processed by conditioning the neural re-rendering network 108 on the expected semantic label map 124 of the output image so that the neural re-rendering network 108 can learn to ignore these transient objects 125 rather than trying to hallucinate their locations. The generation and inclusion of the semantic label map 124 can also effectively discard small or thin scene features (e.g., lamp posts) whose geometry cannot be robustly reconstructed but are easily identified using image segmentation methods. If desired, the neural re-rendering network 108 can also render scenes without people by conditioning it on the semantic tag map 124. This can prevent the neural re-rendering network 108 from encoding changes caused by the position (e.g., appearance vector) of transient objects 125 in the appearance code 142, or associating such transient objects 125 with a particular viewpoint.

[0065] A separate benefit of the semantic map is that it allows the neural re-rendering network 108 to infer static objects in the scene that are not captured in the 3D reconstruction, such as lampposts in the Piazza San Marco. This prevents the neural re-rendering network 108 from arbitrarily introducing such objects, but rather having them appear where they are detected in the semantic label map 124, which may be a significantly simpler task (e.g., in terms of processing power and memory). Additionally, by adding the semantic label map 124 to the image data cache 106, the appearance encoder 138 may be licensed to infer semantic categories such as sky or ground when computing the appearance code 142. Next, during the adjustment training phase 158, the neural network trainer 148 jointly trains (e.g., jointly fine-tunes) the appearance encoder 138 and the neural re-rendering network 108. Staged training may improve capture of scene appearance and scene modeling.

[0066] Fig.11An appearance training phase 152 of a neural network trainer 148 is illustrated according to one aspect. In the appearance training phase 152, the neural re-rendering network 108 is deactivated, wherein only the appearance encoder 138 is trained using the input image 102 and the calibration data set 146 of the image data cache 106. In other words, during the appearance training phase 152, the appearance encoder 138 is isolated from the re-rendering network 108. The training of the appearance encoder 138 generates weights 151 for the appearance encoder 138.

[0067] During the appearance training phase 152, the appearance encoder 138 is configured to receive a pair of the input image 102 and the image data buffer and compute an appearance code 142. In some examples, the appearance code 142 is a latent appearance vector The capture cannot be from the input domain B i The appearance encoder 138 is configured to calculate a latent appearance vector (e.g., appearance code 142) as Where E a The input image 102 (Ii) and the image data buffer (B i ) as input to the appearance encoder 138. In some examples, the appearance encoder 138 is configured to generate an image by converting the input image 102 (I i ) in the lighting and image data cache (B i ) to learn an appearance model based on scene geometry in the neural network. For example, to compute a latent appearance vector (e.g., appearance code 142), the neural network trainer 148 trains the appearance encoder 138 to produce an appearance image of size (W / 2K, H / 2K, Z), where K is the number of subsampling operations in the appearance encoder 138, and Z is the dimensionality of the appearance latent vector. The appearance image is averaged over the spatial dimensions to produce a final latent appearance vector of Z dimensions. In some examples, Z is 8. The latent appearance code is then appended to the feature dimension on the bottleneck layer of the neural re-rendering network 108.

[0068] To train appearance encoder 138 , neural network trainer 148 is configured to calculate triplet loss 164 . Fig.12 and Fig.13An example of calculating the triplet loss 164 during the appearance training stage 152 is illustrated. For example, for each input image 102, the neural network trainer 148 samples a near neighbor image 170 and a far neighbor image 172 using the style loss as a distance metric, and the neural network trainer 148 trains the appearance encoder 138 to optimize the triplet loss 164 therebetween. For example, the distance metric between the input image 102 and the near neighbor image 170 is a small style distance 171 (e.g., a relatively small style loss), and the distance metric between the input image 102 and the far neighbor image 172 is a large style distance 173 (e.g., a relatively large style loss).

[0069] For example, refer to Fig.13 , the neural network trainer 148 performs a proxy task that optimizes the appearance embedding 175 of the input image 102 into the appearance embedding space 174 using a distance metric (e.g., a style loss) between the input images 102. If two images are close in the distance metric (e.g., the input image 102 and the neighbor image 170), their appearance embeddings 175 should also be close in the appearance embedding space 174. In some examples, the distance metric may ignore the content or viewpoint of the input image 102 and the image data cache 106, where the appearance encoder 138 may encode the appearance embedding 175 independent of the viewpoint.

[0070] To train the appearance embedding 175, the neural network trainer 148 uses a triplet loss 164, where for each input image 102, the neural network trainer 148 finds a set of k nearest and farthest neighbor images given by the style loss, from which the neural network trainer 148 can sample positive samples I and I, respectively. p (e.g., neighbor image 170) and negative sample I n (e.g., distant neighbor image 172). The neural network trainer 148 calculates the triplet loss 164 as follows:

[0071] Equation (1):

[0072] in is the Gram matrix of the jth layer activations of the VGG network for image Ii and α is the separation margin.

[0073] Fig.14 An example of a neural network trainer 148 configured to perform a re-rendering training phase 154 is illustrated. After the appearance training phase 152, the neural network trainer 148 is configured to perform a re-rendering training phase 154, wherein the weights 151 of the appearance encoder 138 are held fixed. During the re-rendering training phase 154, the neural network trainer 148 is configured to train the neural re-rendering network 108 using a loss 166 (e.g., a GAN and VGG loss).

[0074] During the re-rendering training phase 154, the neural network trainer 148 provides the image data cache 106 to the neural re-rendering network 108 and transforms the image data cache 106 into a re-rendered image 110 from the same viewpoint as the image data cache 106. In some examples, the neural re-rendering network 108 receives the color 116 and depth 118 of the image data cache 106 and transforms the color 116 and depth 118 into the re-rendered image 110. In some examples, the neural re-rendering network 108 is configured to perform an image-to-image conversion. Then, the neural network trainer 148 is configured to calculate a loss 166 using the re-rendered image 110 and the input image 102. In some examples, the loss 166 includes a perceptual loss and / or an adversarial loss. In some examples, the loss 166 includes a generative adversarial network (GAN) loss and a VGG (perceptual) loss. In some examples, the input image 102 is an original photo, and the original photo is compared to the photo generated by the neural re-rendering network 108, and the difference between the two can be reflected by the loss 166.

[0075] Fig.15 An example of a neural network trainer 148 configured to generate a semantic label map 124 during training of the neural network trainer 148 is illustrated. Additionally, as part of a re-rendering training phase 154, the neural network trainer 148 is configured to perform semantic conditioning 156 to learn and detect transient objects 125. In some examples, during the re-rendering training phase 154, the semantic label map 124 is generated from the input image 102. In some examples, the input image 102 is a ground truth image 102a, where the semantic label map 124 is generated from the ground truth image 102a. During the re-rendering training phase 154, the semantic segmentation network 130 is configured to receive the input image 102 (or the ground truth image 102a) and generate the semantic label map 124. The semantic label map 124, the color 116, and the depth 118 (which form part of the image data cache 106) are then provided to the neural re-rendering network 108.

[0076] In some examples, after training the neural re-rendering network 108 (e.g., during testing or runtime of the system), a novel viewpoint may be generated, wherein (as discussed above) an image data cache 106 having the novel viewpoint is generated from the 3D model 104. However, at this point, the image data cache 106 may not include the semantic label map 124.

[0077] Fig.16An example of generating a semantic label map 124 from a portion of an image data cache 106 according to one aspect is illustrated. For example, an image-to-image conversion network 132 is configured to generate the semantic label map 124 from layers of color 116 and depth 118 of the image data cache 106 to generate the semantic label map 124. The transient object detector 128 includes the image-to-image conversion network 132. Then, when generating a re-rendered image 110 from the image data cache 106, the semantic label map 124 can be used to remove the transient objects 125.

[0078] In some examples, to re-render the virtual camera path, the semantic label map 124 for each frame in the virtual camera path is synthesized. To this end, an image-to-image conversion network 132 is trained that caches the image data 106 (B i ) as input, and given the rendered image data cache 106 (B i ) to estimate the “reasonable” semantic label of the viewpoint In some examples, image-to-image translation network 132 utilizes the same architecture as neural re-rendering network 108 (minus injected appearance code 142) to perform image translation on sample and ground truth images S from calibration dataset 146. i The semantic label map 124 (eg, the ground truth image 102a) is trained and masks the loss on pixels labeled as transient as defined by a curated list of transient object categories.

[0079] After the re-rendering training phase 154 , the neural network trainer 148 is configured to perform an adjustment training phase 158 . Fig.17 An example of a neural network trainer 148 configured to perform an adjustment training phase 158 is illustrated according to one aspect. During the adjustment training phase 158, the neural network trainer 148 is configured to train the appearance encoder 138 and the neural re-rendering network 108 using the triplet loss 164 and the loss 166 as explained above. This staged approach simplifies and stabilizes the training of the appearance encoder 138, enabling the training of simpler networks with fewer regularization terms. In particular, the cycle and cross-cycle consistency losses, the latent vector reconstruction loss, and the KL divergence loss can be removed.

[0080] Fig.18 An example of a neural re-rendering system 100 is shown for generating a re-rendered image 110 according to any appearance condition and / or with (or without) transient objects 125. Fig.18In the example of FIG. 1 , it is assumed that the neural re-rendering network 108 and the appearance encoder 138 have been trained. The cache renderer 136 can receive a viewpoint request specifying a viewpoint and can render the image data cache 106 from the 3D model 104 according to the viewpoint. The neural re-rendering network 108 can also receive the image data cache 106 and the appearance code 142 and generate a re-rendered image 110 with the requested appearance condition. In some examples, the appearance condition need not be the same as a real photo taken from the same vantage point (with a fixed appearance). For example, the appearance condition of the re-rendered image 110 can be different from the appearance condition of the input image 102 (e.g., the corresponding original image) having the same viewpoint as the viewpoint of the re-rendered image 110. For example, the original image can be sunny, but the re-rendered image 110 can be cloudy (or any appearance condition specified by the appearance code 142). In some examples, the appearance code 142 is an interpolation of the appearance codes of the two input images. In some examples, the appearance condition is the same as the original image (e.g., and therefore has the same appearance code 142).

[0081] In some examples, the neural re-rendering network 108 may also receive a transient data deletion request indicating to delete the transient objects 125 from the re-rendered image 110. If the neural re-rendering network 108 receives the transient data deletion request, the neural re-rendering network 108 may use the semantic tag map 124 as a guide to identify the transient objects 125 and then delete them from the re-rendered image 110.

[0082] Fig.19 The diagram depicts a flowchart 1900 depicting example operation of the neural re-rendering system 100 according to one aspect.

[0083] Operation 1902 includes obtaining a three-dimensional (3D) model 104 representing a physical space scene, wherein the 3D model 104 is constructed from a collection 101 of input images 102. Operation 1904 includes rendering an image data cache 106 from the 3D model 104 according to a viewpoint, wherein the image data cache 106 represents an image reconstructed from the 3D model 104. Operation 1906 includes receiving the image data cache 106 by a neural re-rendering network 108. Operation 1908 includes receiving an appearance code 142 specifying an appearance condition by the neural representation network 108. Operation 1910 includes transforming the image data cache 106 by the neural re-rendering network 108 into a re-rendered image 110 having a viewpoint of the image data cache 106 and an appearance condition specified by the appearance code 142. In some examples, the re-rendered image 110 has a higher resolution than a reconstructed image represented by the image data cache 106. In some examples, the re-rendered image 110 has a higher level of detail than the reconstructed image represented by the image data cache 106. In some examples, re-rendered image 110 is a more accurate (or faithful) depiction of the physical space than the reconstructed image represented by image data cache 106 .

[0084] Fig. 20 An example of an example computer device 2000 and an example mobile computer device 2050 that can be used with the techniques described herein are shown. The computing device 2000 includes a processor 2002, a memory 2004, a storage device 2006, a high-speed interface 2008 connected to the memory 2004 and a high-speed expansion port 2010, and a low-speed interface 2012 connected to a low-speed bus 2014 and the storage device 2006. Each of the components 2002, 2004, 2006, 2008, 2010, and 2012 is interconnected using various buses and can be mounted on a general purpose motherboard or otherwise appropriately mounted. The processor 2002 is capable of processing instructions for execution within the computing device 2000, including instructions stored in the memory 2004 or the storage device 2006 to display graphical information for a GUI on an external input / output device (such as a display 2016 coupled to the high-speed interface 2008). In other embodiments, multiple processors and / or multiple buses may be used as appropriate, along with multiple memories and memory types. In addition, multiple computing devices 2000 may be connected, with each device providing a portion of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

[0085] The memory 2004 stores information within the computing device 2000. In one embodiment, the memory 2004 is one or more volatile memory units. In another embodiment, the memory 2004 is one or more non-volatile memory units. The memory 2004 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0086] Storage device 2006 can provide mass storage for computing device 2000. In one embodiment, storage device 2006 can be or include computer readable media, such as floppy disk device, hard disk device, optical disk device, or magnetic tape device, flash memory or other similar solid state storage device, or array of devices, including devices in storage area network or other configurations. Computer program products can be tangibly embodied in information carriers. Computer program products can also include instructions for performing one or more methods as described above when executed. The information carrier is a computer readable medium or machine readable medium, such as memory 2004, storage device 2006 or memory on processor 2002.

[0087] The high-speed controller 2008 manages bandwidth-intensive operations of the computing device 2000, while the low-speed controller 2012 manages less bandwidth-intensive operations. This allocation of functions is exemplary only. In one embodiment, the high-speed controller 2008 is coupled to the memory 2004, the display 2016 (e.g., through a graphics processor or accelerator), and is coupled to a high-speed expansion port 2010 that can accept various expansion cards (not shown). In an embodiment, the low-speed controller 2012 is coupled to the storage device 2006 and the low-speed expansion port 2014. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, for example, through a network adapter.

[0088] As shown, computing device 2000 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 2020, or implemented multiple times in a group of such servers. It can also be implemented as a part of a rack server system 2024. In addition, it can also be implemented in a personal computer such as a laptop computer 2022. Alternatively, components from computing device 2000 can be combined with other components in a mobile device (not shown), such as device 2050. Each such device can contain one or more computing devices 2000, 2050, and the entire system can be composed of multiple computing devices 2000, 2050 that communicate with each other.

[0089] Among other components, computing device 2050 includes processor 2052, memory 2064, input / output devices such as display 2054, communication interface 2066, and transceiver 2068. Storage devices, such as microdrives or other devices, may also be provided to device 2050 to provide additional storage. Each of components 2050, 2052, 2064, 2066, and 2068 are interconnected using various buses, and multiple components may be mounted on a common motherboard or otherwise as appropriate.

[0090] The processor 2052 is capable of executing instructions within the computing device 2050, including instructions stored in the memory 2064. The processor can be implemented as a chipset including a single or multiple analog and digital processor chips. For example, the processor can provide cooperation with other components of the device 2050, such as control of a user interface, applications run by the device 2050, and wireless communications by the device 2050.

[0091] The processor 2052 can communicate with the user through the control interface 2058 and the display interface 2056 coupled to the display 2054. For example, the display 2054 can be a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other appropriate display technology. The display interface 2056 may include appropriate circuits for driving the display 2054 to present graphics and other information to the user. The control interface 2058 can receive commands from the user and convert them to submit to the processor 2052. In addition, an external interface 2062 in communication with the processor 2052 can be provided to enable the device 2050 to communicate with other devices in the near area. For example, the external interface 2062 can provide wired communication in some embodiments, or provide wireless communication in other embodiments, and multiple interfaces can also be used.

[0092] The memory 2064 stores information within the computing device 2050. The memory 2064 can be implemented as one or more computer-readable media, one or more volatile storage units, or one or more non-volatile storage units. An expansion memory 2074 can also be provided and connected to the device 2050 via an expansion interface 2072, for example, the expansion interface 2072 can include a SIMM (single in-line memory module) card interface. Such an expansion memory 2074 can provide additional storage space for the device 2050, or can also store applications or other information for the device 2050. Specifically, the expansion memory 2074 can include instructions for performing or supplementing the above-mentioned processing, and can also include security information. Therefore, for example, the expansion memory 2074 can be provided as a security module for the device 2050, and can be programmed with instructions that permit the secure use of the device 2050. In addition, the security application can be provided via a SIMM card together with the additional information, such as placing the identification information on the SIMM card in an inviolable manner.

[0093] For example, the memory may include flash memory and / or NVRAM memory, as described below. In one embodiment, the computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more of the methods described above. The information carrier is a computer readable medium or machine readable medium such as memory 2064, expansion memory 2074, or memory on processor 2052, which can be received, for example, via transceiver 2068 or external interface 2062.

[0094] Device 2050 can communicate wirelessly via a communication interface 2066, which may include digital signal processing circuitry when necessary. Communication interface 2066 can provide communication under various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, etc. For example, such communication can occur through a radio frequency transceiver 2068. In addition, short-range communication can occur, such as using Bluetooth, Wi-Fi or other such transceivers (not shown). In addition, a GPS (global positioning system) receiver module 2070 can provide additional navigation and location-related wireless data to device 2050, which can be used appropriately by applications running on device 2050.

[0095] Device 2050 may also communicate audibly using an audio codec 2060 that may receive voice information from a user and convert it into usable digital information. Audio codec 2060 may likewise generate audible sounds for the user, such as through a speaker, for example, in an earpiece of device 2050. Such sounds may include sounds from a voice phone call, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications running on device 2050.

[0096] As shown, computing device 2050 can be implemented in many different forms. For example, it can be implemented as a cellular phone 2080. It can also be implemented as part of a smart phone 2082, a personal digital assistant, or another similar mobile device.

[0097] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include embodiments in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be dedicated or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and coupled to send data and instructions to a storage system, at least one input device, and at least one output device. In addition, the term "module" can include software and / or hardware.

[0098] These computer programs (also referred to as programs, software, software applications or codes) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium", "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0099] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) to display information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including sound, voice, or tactile input.

[0100] The systems and techniques described herein can be implemented on a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0101] A computing system can include clients and servers. Clients and servers are generally remote from each other and generally interact through a communication network. The relationship between a client and a server is created by computer programs running on the respective computers and having a client-server relationship to each other.

[0102] In some embodiments, Fig. 20 The computing device depicted in FIG. 20 may include sensors connected to a virtual reality (VR headset 2090). For example, including Fig. 20One or more sensors on the computing device 2050 or other computing device depicted in the VR headset 2090 can provide input to the VR headset 2090, or generally, can provide input to the VR space. Sensors can include, but are not limited to, touch screens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device 2050 can use sensors to determine the absolute position and / or detected rotation of the computing device in the VR space, which can then be used as input to the VR space. For example, the computing device 2050 can be incorporated into the VR space as a virtual object such as a controller, a laser pointer, a keyboard, a weapon, etc. When a user incorporates the computing device / virtual object into the VR space, its position can allow the user to position the computing device in a certain way to view the virtual object in the VR space. For example, if the virtual object represents a laser pointer, the user can manipulate the computing device like an actual laser pointer. The user can move the computing device left and right, up and down, in circles, etc. and use the device in a similar manner to using a laser pointer.

[0103] In some embodiments, one or more input devices included on or connected to the computing device 2050 can be used as input to the VR space. The input device may include, but is not limited to, a touch screen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, headphones or earbuds with input capabilities, a game controller, or other connectable input devices. When the computing device is incorporated into the VR space, a user interacting with the input device included on the computing device 2050 can cause specific actions to occur in the VR space.

[0104] In some implementations, the touch screen of computing device 2050 can be rendered as a touch pad in VR space. A user can interact with the touch screen of computing device 2050. For example, in VR headset 2090, the interactions are rendered as movements on the touch pad rendered in VR space. The rendered movements can control objects in VR space.

[0105] In some embodiments, one or more output devices included on the computing device 2050 can provide output and / or feedback to a user of the VR headset 2090 in the VR space. The output and feedback can be visual, tactile, or audio. The output and / or feedback can include, but are not limited to, vibration, turning one or more lights or strobe lights on and off or blinking and / or flashing, sounding an alarm, playing a ringtone, playing a song, and playing an audio file. The output device can include, but is not limited to, a vibration motor, a vibration coil, a piezoelectric device, an electrostatic device, a light emitting diode (LED), a strobe light, and a speaker.

[0106] In some embodiments, computing device 2050 can appear as another object in a computer-generated 3D environment. User interactions with computing device 2050 (e.g., rotating, shaking, touching a touch screen, sliding a finger on a touch screen) can be interpreted as interactions with objects in VR space. In the example of a laser pointer in VR space, computing device 2050 appears as a virtual laser pointer in a computer-generated 3D environment. As the user manipulates computing device 2050, the user in VR space sees the movement of the laser pointer. The user receives feedback from the interaction with computing device 2050 in VR space on computing device 2050 or VR headset 2090.

[0107] In some implementations, in addition to the computing device, one or more input devices (e.g., a mouse, a keyboard) can be rendered in the computer-generated 3D environment. The rendered input devices (e.g., a rendered mouse, a rendered keyboard) can be used when rendering in the VR space to control objects in the VR space.

[0108] Computing device 2000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device 2050 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are exemplary only and are not meant to limit the embodiments of the inventions described and / or claimed herein.

[0109] A number of embodiments have been described, however, it will be appreciated that various modifications can be made without departing from the spirit and scope of the specification.

[0110] In addition, the logic flows depicted in the accompanying drawings do not require the particular order shown, or the sequential order to achieve the desired results. In addition, other steps can be provided, or steps can be removed from the described flow, and other components can be added to the described system or removed therefrom. Therefore, other embodiments are within the scope of the appended claims.

[0111] While the disclosed inventive concepts include those defined in the appended claims, it should be understood that the inventive concepts may also be defined according to the following embodiments:

[0112] Embodiment 1 is a method of neural re-rendering, which includes obtaining a three-dimensional (3D) model representing a physical space scene, rendering an image data cache from the 3D model according to a viewpoint, receiving the image data cache through a neural re-rendering network, receiving an appearance code representing an appearance condition through the neural re-rendering network, and transforming the image data cache into a re-rendered image having a viewpoint of the image data cache and an appearance condition specified by the appearance code through the neural re-rendering network.

[0113] Embodiment 2 is the method of embodiment 1, wherein the image data cache represents reconstructed images from the 3D model.

[0114] Embodiment 3 is the method of any one of embodiments 1 to 2, wherein a collection of 3D model input images is constructed.

[0115] Embodiment 4 is the method of any one of embodiments 1 to 3, wherein the appearance condition of the re-rendered image is different from the appearance condition of the input image having the same viewpoint as the viewpoint of the re-rendered image.

[0116] Embodiment 5 is the method of any one of embodiments 1 to 4, wherein the resolution of the re-rendered image is higher than the resolution of the reconstructed image (or image data cache).

[0117] Embodiment 6 is the method of any one of embodiments 1 to 5, wherein the method further comprises modifying the appearance code to represent a second appearance condition of the scene, wherein the transformation comprises generating a re-rendered image according to the modified appearance code so that the re-rendered image includes the second appearance condition.

[0118] Embodiment 7 is the method of any one of embodiments 1 to 6, wherein the appearance condition of the re-rendered image is the same as the appearance condition of the input image having the same viewpoint as the viewpoint of the re-rendered image.

[0119] Embodiment 8 is the method of any one of embodiments 1 to 7, wherein the method further comprises receiving, by the neural re-rendering network, a semantic label map indicating a location of a transient object in a scene.

[0120] Embodiment 9 is the method of any one of embodiments 1 to 8, wherein the transforming comprises generating a re-rendered image using the semantic tag map as a guide such that transient objects are deleted from the re-rendered image.

[0121] Embodiment 10 is the method of any one of embodiments 1 to 9, wherein the image data cache includes a semantic tag map.

[0122] Embodiment 11 is the method of any one of embodiments 1 to 10, wherein the method includes receiving a cache of image data via an image-to-image conversion network.

[0123] Embodiment 12 is the method of any one of embodiments 1 to 11, wherein the method includes generating, by an image-to-image translation network, a semantic tag map based on an image data cache.

[0124] Embodiment 13 is the method of any one of embodiments 1 to 12, wherein the image data cache comprises a plurality of layers.

[0125] Embodiment 14 is the method of any of embodiments 1 to 13, wherein the plurality of layers includes a color layer and a depth layer.

[0126] Embodiment 15 is the method of any one of embodiments 1 to 14, wherein the transforming comprises generating a re-rendered image based on the color layer and the depth layer.

[0127] Embodiment 16 is the method of any one of embodiments 1 to 15, wherein the method comprises generating a calibration data set using the 3D model.

[0128] Embodiment 17 is the method of any one of embodiments 1 to 16, wherein the calibration data set comprises a plurality of image pairs.

[0129] Embodiment 18 is the method of any one of embodiments 1 to 17, wherein each pair comprises an input image and an image data buffer rendered from a 3D model.

[0130] Embodiment 19 is the method of any one of embodiments 1 to 18, wherein the input image and the image data cache are from the same viewpoint.

[0131] Embodiment 20 is the method of any one of embodiments 1 to 19, wherein the image data buffer is aligned with the input image pixels.

[0132] Embodiment 21 is the method of any one of embodiments 1 to 20, wherein the method includes generating one or more semantic label maps based on a portion of the calibration dataset.

[0133] Embodiment 22 is the method of any one of embodiments 1 to 21, wherein the semantic tag map indicates locations of transient objects and non-transient objects.

[0134] Embodiment 23 is the method of any one of embodiments 1 to 22, wherein the method comprises training a neural re-rendering network using a calibration dataset and a semantic label map.

[0135] Embodiment 24 is the method of any one of embodiments 1 to 23, wherein the method includes training an appearance encoder to learn an appearance code for a scene using a calibration dataset.

[0136] Embodiment 25 is the method of any one of embodiments 1 to 24, wherein the method comprises training an appearance encoder using a calibration dataset with a triplet loss to learn an appearance code for a scene.

[0137] Embodiment 26 is a system comprising one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, cause the one or more computers to perform the method of any one of embodiments 1 to 25.

[0138] Embodiment 27 is a computer storage medium encoded with a computer program, wherein the program includes operable instructions that, when executed by a data processing device, cause the data processing device to perform the method of any one of embodiments 1 to 25.

[0139] Embodiment 28 is a non-transitory computer-readable medium storing executable instructions, which, when executed by at least one processor, are configured to cause the at least one processor to obtain a three-dimensional (3D) model representing a physical space scene, render an image data cache from the 3D model according to a viewpoint, receive the image data cache by a neural re-rendering network, receive an appearance code representing an appearance condition by the neural re-rendering network, and transform the image data cache into a re-rendered image through the neural re-rendering network, so that the re-rendered image has the appearance condition specified by the appearance code and the viewpoint of the image data cache.

[0140] Embodiment 29 is the non-transitory computer readable medium of embodiment 28, wherein the image data cache represents a reconstructed image from the 3D model.

[0141] Embodiment 30 is the non-transitory computer readable medium of any one of embodiments 28 to 29, wherein the 3D model is constructed from a collection of input images.

[0142] Embodiment 31 is the non-transitory computer-readable medium of any one of embodiments 28 to 30, wherein the executable instructions include instructions to cause at least one processor to receive a semantic tag map through a neural re-rendering network, the semantic tag map indicating the location of transient objects in the scene.

[0143] Embodiment 32 is a non-transitory computer-readable medium of any one of embodiments 28 to 31, wherein the executable instructions include instructions to cause at least one processor to generate a re-rendered image using a semantic tag map as a guide through a neural re-rendering network, such that transient objects are deleted from the re-rendered image.

[0144] Embodiment 33 is the non-transitory computer-readable medium of any one of embodiments 28 to 32, wherein the image data cache includes a semantic tag map.

[0145] Embodiment 34 is a non-transitory computer-readable medium of any one of embodiments 28 to 33, wherein the executable instructions include instructions for causing at least one processor to receive an image data cache through an image-to-image conversion network, and to generate, by the image-to-image conversion network, a semantic tag mapping based on the image data cache.

[0146] Embodiment 35 is the non-transitory computer-readable medium of any one of embodiments 28 to 34, wherein the executable instructions include instructions for causing the at least one processor to receive, via a semantic segmentation network, a ground truth image associated with an image data cache.

[0147] Embodiment 36 is the non-transitory computer-readable medium of any one of embodiments 28 to 35, wherein the executable instructions cause at least one processor to generate a semantic label map based on a ground truth image through a semantic segmentation network.

[0148] Embodiment 37 is the non-transitory computer readable medium of any one of embodiments 28 to 36, wherein the image data cache comprises a plurality of layers.

[0149] Embodiment 38 is the non-transitory computer readable medium of any one of embodiments 28 to 37, wherein the plurality of layers comprises a color layer and a depth layer.

[0150] Embodiment 39 is the non-transitory computer-readable medium of any one of embodiments 28 to 38, wherein the instructions to transform include instructions to generate a re-rendered image based on the color layer and the depth layer.

[0151] Embodiment 40 is the method according to any one of embodiments 28 to 39.

[0152] Embodiment 41 is a system comprising one or more computers and one or more storage devices storing operable instructions which, when executed by the one or more computers, cause the one or more computers to perform the operations of any one of Embodiments 28 to 39.

[0153] Embodiment 42 is a neural re-rendering system comprising a three-dimensional (3D) generator configured to generate a 3D model representing a physical space scene based on a collection of input images; and a neural re-rendering network configured to transform an image data cache rendered from the 3D model to a re-rendered image.

[0154] Embodiment 43 is the neural re-rendering system of embodiment 42, further comprising an appearance encoder configured to encode multiple appearance conditions of the scene into an appearance code.

[0155] Embodiment 44 is the neural re-rendering system of any one of embodiments 42 to 43, wherein the neural re-rendering network is configured to receive the appearance code such that modification of the appearance code causes the rendered image to be generated according to different appearance conditions.

[0156] Embodiment 45 is the neural re-rendering system of any one of embodiments 42 to 44, further comprising a neural network trainer configured to train the neural re-rendering network and the appearance encoder using the training data.

[0157] Embodiment 46 is the neural re-rendering system of any one of embodiments 42 to 45, wherein the neural network trainer is configured into a neural re-rendering network and an appearance encoder according to a plurality of training stages.

[0158] Embodiment 47 is the neural re-rendering system of any one of embodiments 42 to 46, wherein the plurality of training phases comprises an appearance training phase, wherein an appearance encoder is trained.

[0159] Embodiment 48 is the neural re-rendering system of any one of embodiments 42 to 47, wherein the plurality of training phases includes a re-rendering training phase in which the neural re-rendering network is trained.

[0160] Embodiment 49 is a neural re-rendering system according to any one of embodiments 42 to 48, wherein the multiple trainings include a tuning training phase of jointly training the appearance encoder and the neural re-rendering network.

[0161] Embodiment 50 is the neural re-rendering system of any one of embodiments 42 to 49, wherein the 3D model is a point cloud generated by structure from motion (Sfm) and multi-view stereo (MVS) techniques.

[0162] Embodiment 51 is the neural re-rendering system of any one of embodiments 42 to 50, wherein the image data cache comprises a plurality of layers, the plurality of layers comprising a color layer, a depth layer, and a semantic tag map layer indicating locations of transient objects.

[0163] Embodiment 52 is a neural re-rendering system according to any one of embodiments 42 to 51, wherein the neural re-rendering network is a first neural network and the appearance encoder is a second neural network.

[0164] Embodiment 53 is a method comprising the operation of the neural re-rendering system of any of embodiments 42 to 52.

[0165] Embodiment 54 is a computer storage medium encoded with a computer program, the program comprising operable instructions that, when executed by a data processing device, cause the data processing device to perform the operations of the neural re-rendering system of any one of embodiments 42 to 52.

Claims

1. A method for neural re-rendering, the method comprising: Obtaining a three-dimensional 3D model representing a physical space scene, the 3D model being constructed from a collection of input images; The appearance encoder encodes the multiple appearance conditions of the physical space into appearance codes; rendering an image data cache from the 3D model according to a viewpoint, the image data cache representing an image reconstructed from the 3D model; generating a semantic label map using the image data cache or an input image corresponding to the image data cache; and generating, by a neural re-rendering network, a re-rendered image using the image data cache, the semantic label map, and one or more values ​​of the appearance code as inputs to the neural re-rendering network, such that the re-rendered image has the viewpoint of the image data cache and an appearance condition from among the plurality of appearance conditions, The appearance encoder is trained to estimate the appearance code using an image data cache and an input image, and the neural re-rendering network is trained to estimate a re-rendered image using a semantic label map and the appearance code from the appearance encoder.

2. The method according to claim 1, wherein: The appearance condition of the re-rendered image is different from the appearance condition of the input image, the input image having the same viewpoint as the viewpoint of the re-rendered image.

3. The method according to claim 1, wherein: The plurality of appearance conditions are encoded by correlating the lighting in each input image with the scene geometry in a corresponding image data buffer such that the appearance code is viewpoint independent.

4. The method according to claim 1, wherein: The appearance encoder is trained to encode the plurality of appearance conditions without using the neural re-rendering network during at least a portion of the time.

5. The method according to claim 1, further comprising: receiving, by an image-to-image conversion network, the image data cache; as well as The semantic label map is generated by the image-to-image translation network based on the color layer and the depth layer of the image data cache.

6. The method according to claim 1, wherein: The image data cache includes a plurality of layers, the plurality of layers including a color layer and a depth layer, wherein the color layer and the depth layer are used as input to the neural re-rendering network.

7. The method according to any one of claims 1 to 6, further comprising: generating a calibration dataset using the 3D model, the calibration dataset comprising a plurality of image pairs, each image pair comprising an input image and a corresponding image data cache rendered from the 3D model, the input image and the corresponding image data cache being from the same viewpoint, the corresponding image data cache of each image pair being pixel-calibrated with the input image; generating one or more semantically labeled maps based on a portion of the calibration data set, wherein the one or more semantically labeled maps indicate locations of objects predicted to move out of the physical space; and wherein the neural re-rendering network is trained using the calibration dataset, the one or more semantic tag maps, and the appearance code from the appearance encoder.

8. The method according to claim 7, wherein: The appearance encoder is trained using the calibration dataset and for a period of time without using the neural re-rendering network.

9. The method according to claim 1, further comprising: A transient data omission request is received indicating to omit at least one object predicted to move out of the physical space, wherein the re-rendered image is generated without the at least one object predicted to move out of the physical space.

10. A non-transitory computer-readable medium storing executable instructions, which, when executed by at least one processor, cause the at least one processor to perform operations comprising: Obtaining a three-dimensional 3D model representing a physical space scene, the 3D model being constructed from a collection of input images; The appearance encoder encodes the multiple appearance conditions of the physical space into appearance codes; rendering an image data cache from the 3D model according to a viewpoint, the image data cache representing an image reconstructed from the 3D model; generating a semantic label map using the image data cache or an input image corresponding to the image data cache; and generating, by a neural re-rendering network, a re-rendered image using the image data cache, the semantic label map, and one or more values ​​of the appearance code, such that the re-rendered image has the viewpoint of the image data cache and an appearance condition from among the plurality of appearance conditions, The appearance encoder is trained to estimate the appearance code using an image data cache and an input image, and the neural re-rendering network is trained to estimate a re-rendered image using a semantic label map and the appearance code from the appearance encoder.

11. The non-transitory computer readable medium of claim 10, wherein: The semantic label map indicates locations of objects predicted to move out of the physical space.

12. The non-transitory computer readable medium of claim 10, wherein: The image data cache comprises a deferred shading cache or a geometry cache, the image data cache comprises a color layer and a depth layer, the image data cache comprises the semantic tag map.

13. The non-transitory computer readable medium of claim 10, wherein: The operations also include: receiving, by an image-to-image translation network, the image data buffer; and The semantic label map is generated based on the image data cache by the image-to-image translation network, the image-to-image translation network comprising a neural network different from the appearance encoder and the neural re-rendering network.

14. The non-transitory computer readable medium of claim 10, wherein: The operations also include: receiving, by a semantic segmentation network, a ground truth image corresponding to the image data cache; and The semantic label map is generated by the semantic segmentation network based on the ground truth image.

15. The non-transitory computer readable medium of any one of claims 10 to 14, wherein: The image data cache comprises a plurality of layers, the plurality of layers comprising a color layer, a depth layer, and a surface normal layer, wherein at least the color layer and the depth layer are used as input to the neural re-rendering network.

16. A neural re-rendering system comprising: a three-dimensional 3D generator, the 3D generator being configured to generate a 3D model representing a physical space scene based on a collection of input images; a neural re-rendering network configured to generate a re-rendered image using a cache of image data rendered from the 3D model and a semantic label map as input to the neural re-rendering network; an appearance encoder configured to encode a plurality of appearance conditions of the scene into appearance codes, the neural re-rendering network configured to receive one or more values ​​of the appearance codes, such that the re-rendered image is generated using appearance conditions of the plurality of appearance conditions; as well as a neural network trainer configured to train the appearance encoder to estimate the appearance code using an image data cache and an input image, the neural network trainer configured to train the neural re-rendering network to estimate a re-rendered image using a semantic label map and the appearance code from the appearance encoder.

17. The neural re-rendering system of claim 16, wherein: The semantic label map indicates locations of objects predicted to move out of the physical space.

18. The neural re-rendering system of claim 16, wherein: The neural network trainer is configured to train the neural re-rendering network and the appearance encoder according to a plurality of training phases, the plurality of training phases comprising: an appearance training phase in which the appearance encoder is trained without using the neural re-rendering network, a re-rendering training phase in which the neural re-rendering network is trained without changing weights of the appearance encoder, and an adjustment training phase in which the appearance encoder and the neural re-rendering network are jointly trained.

19. The neural re-rendering system of claim 16, wherein: The 3D model is a point cloud generated by structure from motion and multi-view stereo techniques.

20. The neural re-rendering system of claim 16, wherein: The image data cache includes a plurality of layers, each of the plurality of layers providing information on a per-pixel basis, the plurality of layers including a color layer, a depth layer, and a surface normal layer.

Images