Real-time view synthesis

Abstract

1. A method comprising: receiving a plurality of two-dimensional (2D) images representing frames of a streaming three-dimensional (3D) video; generating a plurality of meshes corresponding to one of the plurality of 2D images; generating a composite mesh based on the plurality of meshes; generating a left-eye 3D image and depth map based on the composite mesh; and generating a right-eye 3D image and depth map based on the composite mesh, wherein the left-eye 3D image and depth map and the right-eye 3D image and depth map have a viewpoint perspective based on the receiver of the streaming 3D video.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 63 / 383,866, filed November 15, 2022, the disclosure of which is incorporated herein by reference in its entirety.

[0002] Embodiments relate to rendering three-dimensional left-eye and right-eye images. [Background technology]

[0003] Generating a three-dimensional (3D) image from multiple two-dimensional (2D) images can involve stitching the 2D images. The stitching operation can include simultaneously calculating all possible translations (x, y, z) between two 2D images relative to the 3D field of view. By calculating the translations, the best overlap in terms of a cross-correlation measure can be determined. When more than two input images are used, the exact placement of portions of the images (sometimes called tiles) can be globally optimized (e.g., the resulting 3D image is modified to remove gaps and overlaps). Summary of the Invention

[0004] An exemplary embodiment describes a neural rendering and view synthesis system configured to synthesize two viewpoints (e.g., left eye and right eye) based on the eye positions of a viewer at the receiver side of a streaming sequence of 3D images. The 3D images may be synthesized as a layered mesh based on multiple 2D images and rendered before streaming the sequence of 3D images. The layered mesh may be used to render any potential viewpoint perspective of a user viewing the 3D images using a playback device.

[0005] The exemplary embodiments will be more fully understood from the following detailed description herein and the accompanying drawings, in which like elements are indicated with like reference numerals and are given by way of example only, and therefore not by way of limitation of the exemplary embodiments. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 illustrates a block diagram of a streaming pipeline, according to an exemplary embodiment. [Figure 2] 1 illustrates a block diagram of a view synthesis system, according to an exemplary implementation. [Figure 3] FIG. 1 illustrates a block diagram of a neural rendering system, according to an exemplary implementation. [Figure 4A] FIG. 1 illustrates a block diagram of an exemplary machine learning downsampling network, according to an exemplary implementation. [Figure 4B] 1 illustrates a block diagram of an exemplary machine learning view synthesis network, according to an exemplary implementation. [Figure 4C] FIG. 1 illustrates a block diagram of an exemplary machine learning upsampling network, according to an exemplary implementation. [Figure 5] 1 illustrates a block diagram of a system according to an exemplary embodiment. [Figure 6] 1 illustrates a block diagram of a method, according to an exemplary embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0007] It should be noted that these figures are intended to illustrate general features of methods and / or structures utilized in certain exemplary embodiments and to supplement the written description provided below. However, these figures are not to scale, may not precisely reflect the precise structural or performance characteristics of any given embodiment, and should not be construed as defining or limiting the range of values ​​or characteristics encompassed by the exemplary embodiments. For example, the arrangement of modules and / or structural elements may be reduced or exaggerated for clarity. The use of similar or identical reference numbers in various figures is intended to indicate the presence of similar or identical elements or features.

[0008] Generating a 3D image by stitching multiple 2D images is not fast enough for real-time streaming of 3D images. In other words, capturing and stitching 2D images in real-time 3D streaming applications is too slow to provide the desired user experience. Existing solutions may reduce the resolution (e.g., number of pixels) to a very low resolution and still not achieve the frame rate, or frames per second (fps), desired for real-time streaming of 3D images.

[0009] Exemplary implementations can use the trained machine learning model to synthesize a 3D mesh from multiple 2D images. Exemplary implementations can further use the trained machine learning model to render the synthesized 3D mesh to generate two 3D images, each with a perspective field of view (e.g., left eye and right eye). The 3D images can then be streamed to a 3D playback device. Exemplary implementations can stream 3D images at a sufficiently high resolution (e.g., 4K) and frame rate (e.g., 30 fps) to provide a desired user experience in real-time 3D streaming applications.

[0010] Figure 1 shows a block diagram of a 3D streaming pipeline, according to an example implementation. As shown in Figure 1, the 3D streaming pipeline 110 can include a mesh synthesis module 115 and a rendering module 120. The 3D streaming pipeline 110 can be configured to receive multiple 2D images 105 and generate two 3D images that can be streamed to a playback device 125.

[0011] Multiple cameras (e.g., a camera rig) can be configured to capture multiple 2D images 105. In an exemplary implementation, the multiple cameras (e.g., six cameras) can be rolling-shutter RGB cameras that are time-synchronized and share exposure and white balance settings. The multiple 2D images 105 can represent image frames. In an exemplary implementation, the multiple cameras may not capture depth. Thus, the multiple 2D images 105 may not include depth information. In an exemplary implementation, the 3D streaming pipeline 110 can be on the same device (e.g., a transmitting station) as the multiple cameras. In this exemplary implementation, the multiple 2D images 105 can be processed inline by the 3D streaming pipeline 110 as the frame(s) are captured.

[0012] In an exemplary implementation, the 3D streaming pipeline 110 can reside on a different device (e.g., a server) than the multiple cameras. In this exemplary implementation, once the frame(s) are captured, the multiple 2D images 105 can be compressed using the HEVC (h.265) standard, for example, at 25 Mbps per camera, and then communicated to the server for processing by the 3D streaming pipeline 110. In this implementation, the multiple 2D images 105 can be decompressed and processed by the 3D streaming pipeline 110.

[0013] The mesh synthesis module 115 can be configured to synthesize or fuse multiple 2D images 105 into a 3D representation of the scene, sometimes referred to as a layered mesh (e.g., layered mesh 20, described in more detail below), and the rendering module 120 can be configured to render the layered mesh as a 3D image representation of the scene. In an exemplary implementation, the layered mesh can represent a complete 3D scene based on the multiple 2D images 105. In other words, the layered mesh is not configured to render almost any particular viewpoint (e.g., left eye or right eye) of a user viewing the playback device 125. Furthermore, the layered mesh can be used to render any potential viewpoint of a user viewing the playback device 125. Furthermore, the layered mesh can be used to render any potential viewpoint of a user viewing the playback device 125. In other words, the synthesized layered mesh can represent the field of view of any view corresponding to any head position, so that when displayed on the playback device 125, the left-eye and right-eye images rendered based on the synthesized layered mesh can have a field of view that can be changed in six degrees of freedom (DoF) based on the user's field of view and / or head position.

[0014] Rendering module 120 can be configured to render two images and generate two depth maps (e.g., one for each eye of a user viewing playback device 125). The two images can be RGB and depth map views. In an exemplary implementation, playback device 125 can communicate a current or most recent viewpoint field of view and / or head pose of a user viewing playback device 125. Accordingly, rendering module 120 can be configured to render two images and generate two depth maps (e.g., RGB and depth map views) based on the user's current or most recent viewpoint field of view and / or head pose. The rendered images and generated depth maps can be streamed (e.g., communicated) to playback device 125, for example, at 4K resolution and 30 fps.

[0015] In an exemplary implementation, playback device 125 can be configured to perform a last-second reprojection of the rendered image and generated depth map using the user's most recent estimated field of view and / or head pose before rendering to the display of playback device 125. The reprojection can adjust for user movement (e.g., changes in field of view and / or head pose) during the streaming process (e.g., due to system and / or streaming latency).

[0016] FIG. 2 shows a block diagram of a view synthesis system according to an exemplary embodiment. The view synthesis system can be configured to blend image weights and densities and reconstruct depth layers in the form of a layered mesh representation. The view synthesis system can be based on the DeepView algorithm. For example, the view synthesis system can be configured to generate a layered mesh output through a process of iterative refinement. As shown in FIG. 2, the mesh synthesis module 115 includes a rectification module 205, a downsampling module 210, a synthesis module 215, and an upsampling module 220.

[0017] The rectification module 205 can be configured to generate the rectified images 5. In an exemplary implementation, the rectification module 205 can be configured to reproject each of the images 105 onto a layered mesh plane as a rectified image 5. In an exemplary implementation, the layered mesh plane can be a proximal clipping plane. The rectification module 205 can be configured to reduce the resolution of each of the images 105 during the reprojection of each of the images 105. The reprojection of each of the images 105 can ensure, or help to ensure, that image coordinates are consistent across each of the rectified images 5.

[0018] The downsampling module 210 can be configured to downsample each of the rectified images 5, e.g., by a factor of eight (8x), using, e.g., a trained machine learning downsampling network. The machine learning downsampling network can be configured to generate a low-resolution feature map 10 for each of the rectified images 5.

[0019] 4A illustrates a block diagram of an exemplary machine learning downsampling network according to an exemplary implementation as an exemplary element (or implementation) of the downsampling module 210. The machine learning downsampling network 405 may include multiple convolutional layers 410-1, 410-2, 410-3, 410-4, 410-5, 410-6, and 410-7. For example, the machine learning downsampling network 405 may include a series of strided convolutional layers. Thus, the convolutional layers 410-1, 410-2, 410-3, 410-4, 410-5, 410-6, and 410-7 may be strided convolutional layers. Spanning a convolutional layer indicates the number of pixels that the filter matrix of the convolutional layer moves across the input image. The stride length of the convolutional layer indicates the number of steps performed when sliding the filter matrix across the image. In some implementations, the stride length of convolutional layers 410-1, 410-2, 410-3, 410-4, 410-5, 410-6, and 410-7 can be 1 or 2. For example, convolutional layers 410-1, 410-3, 410-5, and 410-7 can have a stride length of 1, and convolutional layers 410-2, 410-4, and 410-6 can have a stride length of 2.

[0020] The multiple convolutional layers 410-1, 410-2, 410-3, 410-4, 410-5, 410-6, 410-7 may be configured to reduce the resolution of the rectified image 5. For example, the multiple convolutional layers 410-1, 410-2, 410-3, 410-4 may be configured to reduce the resolution of the rectified image 5 by two times (2x), the convolutional layers 410-5, 410-6 may be configured to reduce the resolution of the rectified image 5 by two times (2x), and the convolutional layer 410-7 may be configured to reduce the resolution of the rectified image 5 by two times (2x), resulting in a total reduction in resolution of eight times (8x). In some implementations, convolutional layers 410-1, 410-2, 410-3, 410-4, 410-5, 410-6, 410-7 can be configured to increase the channel count of each of rectified images 5, for example, from 4 to 32 channels.

[0021] Referring back to FIG. 2, the synthesis module 215 can be configured to generate the feature layering mesh 15. The feature layering mesh 15 can be a low-resolution layering mesh. In an exemplary implementation, the feature layering mesh 15 can include 288×184 pixels×16 layers. FIG. 4B shows a block diagram of an exemplary machine learning view synthesis network according to an exemplary implementation as an exemplary element (or implementation) of the synthesis module 215.

[0022] As shown in FIG. 4B , in an exemplary embodiment, layers of the feature layering mesh 15 can be initialized to have flat geometries, and the feature map 10 can be projected onto these layers to generate a planar swept volume (PSV). Next, according to an exemplary embodiment, an initialization network (e.g., image-to-layer transition 415-1 and neural network 420-1 (e.g., a convolutional neural network (CNN))) can be configured to calculate an initial estimate of the feature layering mesh 15 (e.g., the output of the neural network 420-1 or layering mesh) based on the PSV. At this point, the layers of the feature layering mesh 15 can include network features (having 32 channels). In an exemplary embodiment, the first 30 channels of the machine learning view synthesis network can include abstract network features (e.g., because the features do not have a specific physical interpretation, the network is free to use the features of the feature layering mesh 15 in any useful way). However, in an exemplary embodiment, the last two channels of the feature layering mesh 15 can be used by the machine learning view synthesis network to derive depth and density information. Thus, machine learning view synthesis networks can be configured to generate (or learn to generate) features for learning depth and density information.

[0023] The feature layer mesh 15 generated by the initialization network can be refined through two successive update steps. The first update step (Update 1) can include layer-to-image transitions 425-1, image-to-layer transitions 415-2, a neural network 420-2 (e.g., a CNN), a visibility component 430-1, and an activation block 435-1. The second update step (Update 2) can include layer-to-image transitions 425-2, image-to-layer transitions 415-3, a neural network 420-2 (e.g., a CNN), a visibility component 430-2, and an activation block 435-2. During each update step, the current feature layer mesh 15 can be backprojected into the input feature map 10. In some implementations, the current feature layer mesh 15 can be compared to determine how well it approximates the actual image captured by the input camera. However, the feature layer mesh 15 backprojected into the input feature map 10 can be used in any useful process. In an exemplary implementation, when projecting features of the feature layered mesh 15 onto the viewpoint of the input feature map 10, geometry derived from the penultimate depth channel of the layered mesh can be used. This channel can be activated with a tanh nonlinearity (e.g., activation blocks 435-1, 435-2, 435-3), scaled by the layer width, and added to a set of depth anchors equally spaced in disparity (e.g., the output of activation blocks 435-1, 435-2, 435-3). Constructing geometry using this technique allows each layer to have its own disparity band, thereby preventing layer overlap in the feature layered mesh 15.

[0024] The layer geometry can then be used to warp from layer space to view space (and back again). While in view space, the latest channels (e.g., density features) of the feature layered mesh 15 can be used to perform compositing operations that help communicate visibility information across layers. These visibility components 430-1, 430-2, 430-3 can be used to reason about occlusion and understand dependencies across layers, so they can be used to aid the update network. The visibility components 430-1, 430-2, 430-3 can be accumulated over the net transmittance to include the net transmittance. The accumulated may include a reconstruction of the scene from behind a plane, and the net transmittance may be a soft occlusion mask of the plane.

[0025] The computation of the visibility components 430-1, 430-2, and 430-3 can improve the functionality of the machine learning view synthesis network because the computation of the visibility components can be the time when information is communicated between layers. To complete the update step, the visibility components 430-1, 430-2, and 430-3 can be warped back from their respective input view spaces to a central layered mesh representation, and then these features can be input into the update network. The update network can be configured to generate deltas that can be added via residual connections to the layered mesh computed in the previous iteration. This allows the feature layered mesh 15 to be generated iteratively (e.g., in multiple update steps) based on the feature map 10. Furthermore, the activation block 435-3, the layer to image 425-3, and the visibility component 430-3 can collectively generate the gradient computation 445. Alternatively (or additionally), the layer to image 425-3 can calculate the visibility component 430-3 (as gradient calculation 445) based on the feature layered mesh 15 and the depth calculated by the activation block 435-3. This rebuild, check, and then refine strategy implemented in the update step can be repeated several times, and the strategy can function like an iterative optimization algorithm. Convergence to a high-quality solution can occur in a few iterations (e.g., three).

[0026] Referring back to FIG. 2 , the upsampling module 220 can be configured to generate the layered mesh 20. In an exemplary implementation, the upsampling module 220 can be configured to increase the resolution of the feature layered mesh 15. In an exemplary implementation, the upsampling module 220 can be configured to increase the density 30 in resolution, for example, by eight times (8x). For example, the density 30 of the feature layered mesh 15 can be increased to a resolution of 1080p. In an exemplary implementation, the upsampling module 220 can be configured to refine the blending weights 25 of the feature layered mesh 15. However, the blending weights 25 and mesh vertices 35 of the layered mesh 20 can remain at a low resolution. Keeping the blending weights 25 and mesh vertices 35 at a low resolution can increase efficiency. For example, the final 3D image may be insensitive to the blending weights 25 and the resolution of the mesh geometry. In contrast, the final 3D image may be sensitive to the alpha and RGB resolution.

[0027] 4C illustrates a block diagram of an exemplary machine learning upsampling network according to an exemplary implementation as an exemplary component (or implementation) of the upsampling module 220. As shown in FIG. 4C, in an exemplary implementation, the machine learning upsampling network 480 can use the feature layered mesh 15 calculated by the view synthesis network 440 and the final set of visibility components (gradient calculation 445), which it then processes using a series of convolutions 450-1, 450-2, 450-3, 450-4, 450-5, 450-6, 450-7, 450-8, 450-9, 450-10, and 450-11, concatenations 455-1, 455-2, and a squeeze and excitation network (including softmax 465 with features and weights, multiplication elements, and addition elements) to estimate the low-resolution blend weights 25, the mesh's vertex 35 positions, and the higher resolution density 30 layers of the layered mesh 20. In some implementations, convolutions 450-7, 450-8 can be referred to as blend models, and convolutions 450-9, 450-10, 450-11 can be referred to as density models. In an exemplary implementation, the above density 30 layers can be upsampled via a depth2space transform 470. In some implementations, the feature layered mesh 15 (e.g., the penultimate channel of the feature layered mesh 15) can be activated and transformed (475) to generate a tensor containing the 3D mesh vertex 35 positions.

[0028] FIG. 3 illustrates a block diagram of a neural rendering system according to an exemplary embodiment. As shown in FIG. 3, the rendering module 120 can be configured to generate a left-eye (LE) image 25 and a right-eye (RE) image 30 based on a layered mesh 30. In an exemplary embodiment, the LE image 25 and the RE image 30 can each be a 4k RGB-plus-depth image. As shown in FIG. 3, the rendering module 120 can include an RE projection module 305, an LE projection module 310, an RE blending module 315, an LE blending module 320, an RE over-compositing module 325, and an LE over-compositing module 330. In an exemplary embodiment, the layered mesh 20 can include 16 mesh layers, each including associated blending weights and density values. Furthermore, the layered mesh 20 can be used to render any potential viewpoint of a user viewing the playback device 125. In other words, the synthesized layered mesh can represent the field of view of any view corresponding to any head position, so that when displayed on the playback device 125, the left-eye and right-eye images rendered based on the synthesized layered mesh can have a field of view that can be changed in six degrees of freedom (DoF) based on the user's field of view and / or head position.

[0029] In an exemplary implementation, each mesh layer of layered mesh 20 can be rasterized into the image space of the rendered output view. This process returns the intersection triangle index and barycentric coordinates of each pixel in the output view. In an exemplary implementation, this information can be used to (1) look up a set of six blending weights (e.g., using barycentric interpolation of the model-generated mesh layer blending weights), (2) look up density values ​​(e.g., using barycentric interpolation of the model-generated mesh layer densities), and (3) calculate the 3D coordinates of the intersection points. These 3D coordinates can be back-projected onto the image plane of the original high-resolution input view to determine RGB values ​​from each input image (e.g., using bilinear interpolation).

[0030] In an exemplary implementation, blending weights may be activated using a softmax nonlinearity and then used to calculate a simple weighted average of RGB values ​​from the input view. Density values ​​may be activated via a softmax nonlinearity and converted to alpha values. In an exemplary implementation, rasterizing a layer to an output view may generate an RGB-plus-alpha value for each pixel in the layer.

[0031] This sequence of rasterization, projection, and sampling steps includes the signal flow shown in Figure 3. For example, the signal flow includes projecting an input image onto a mesh geometry (RE Projection module 305 and LE Projection module 310), blending the meshes together using blend weights, combining with alpha values ​​from the layers, and then projecting the result onto an output view (RE Blend module 315 and LE Blend module 320).

[0032] In an exemplary implementation, the nonlinear activation can include post-resampling activation. In other words, the nonlinear activation can include projecting the input image, blending weights, and densities to the output viewpoint. When the network is trained through a differentiable version of the rendering process described above, the network can learn to expect these activations to occur in a higher-resolution space, and the network can actually learn to utilize the post-interpolation activations to generate clearer, higher-resolution images even when working with relatively low-resolution blending weights and alpha densities. In an exemplary implementation, the post-resampling activation can be referred to as deferred rendering or deferred sampling of the layered mesh. Deferred rendering or deferred sampling of the layered mesh can help achieve high-quality 4k output, even when the layered mesh includes 1080p densities and low-resolution (e.g., 135x240 pixel) geometry and blending weights.

[0033] In an exemplary implementation, rasterization can be repeated for each of the 16 mesh layers of layered mesh 20, and the resulting RGB-plus-alpha layers can then be rendered using alpha compositing to generate a final RGB image for a particular output viewpoint (RE over compositing module 325 and LE over compositing module 330). In an exemplary implementation, RE over compositing module 325 and LE over compositing module 330 can be configured to generate a depth channel by replacing the RGB of each layer with the parallax of that layer and then compositing the parallax-plus-alpha layer.

[0034] In an exemplary implementation, deferred rendering or deferred sampling techniques have the advantage of separating the resolution of the input view, the network output, and the final RGB-plus-depth rendered result. This is advantageous because this separation allows the resolution of each component to be adjusted independently, which is useful for trading off quality against performance. For example, the speed of an RGB-plus-depth renderer can be increased by using a 4k input but outputting RGB-plus-depth images at 1440p.

[0035] The layered mesh 20 described above can be used to represent the 3D structure of a scene. Additionally, the layered mesh provides a mesh representation that can be rasterized to generate an RGB-plus-depth view used for streaming to a playback device 125. Layered meshes can be associated with multi-planar images (MPIs). Using both MPI and layered meshes, viewpoints can be rendered using a combination of view projection and alpha compositing. Mesh layers can occupy disparity bands that can be evenly spaced in disparity (1 / z) space, similar to MPI planes. A view can be rendered by first projecting a layer onto the output viewpoint and then alpha compositing from back to front. However, unlike the planar geometry of MPI, mesh layers may have network-generated geometry that molds itself to the shape of the object being reconstructed. This allows layered meshes to achieve similar quality to MPI with far fewer layers. Layered meshes enable an efficient (10x faster than using MPI) method of learning upsampling and rendering, so 30 fps at high resolutions can be achieved using layered meshes.

[0036] FIG. 5 illustrates a block diagram of a system according to an exemplary implementation. In the example of FIG. 5, the system (e.g., wearable device 300, an augmented reality system, a virtual reality system, a companion device, etc.) may include a computing system or at least one computing device, and it should be understood to represent virtually any computing device configured to implement the techniques described herein. Thus, the device may be understood to include various components that may be utilized to implement the techniques described herein, or different or future versions thereof. By way of example, the system may include a processor 505 and memory 510 (e.g., non-transitory computer-readable memory). The processor 505 and memory 510 may be coupled (e.g., communicatively coupled) by a bus 515.

[0037] The processor 505 may be utilized to execute instructions stored in at least one memory 510. As such, the processor 505 may perform various features and functions described herein, or additional or alternative features and functions. The processor 505 and the at least one memory 510 may be utilized for various other purposes. For example, the at least one memory 510 may represent examples of various types of memory and associated hardware and software that may be used to implement any one of the modules described herein.

[0038] The at least one memory 510 may be configured to store data and / or information related to the device. The at least one memory 510 may be a shared resource. Thus, the at least one memory 510 may be configured to store data and / or information related to other elements in a larger system (e.g., image / video processing or wired / wireless communication). The processor 505 and the at least one memory 510 may be utilized together to perform the techniques described herein. Thus, the techniques described herein may be implemented as code segments (e.g., software) stored on the memory 510 and executed by the processor 505. Thus, the memory 510 may include any combination of the mesh synthesis module 115 and the rendering module 120. The exemplary implementation shown in FIG. 5 is merely one example of a hardware configuration. In other implementations, operations may be shared between computing devices.

[0039] Example 1 FIG. 6 shows a block diagram of a method according to an exemplary embodiment. As shown in FIG. 6, in step S605, a plurality of two-dimensional (2D) images representing frames of a streaming three-dimensional (3D) video are received. In step S610, a plurality of meshes corresponding to the plurality of 2D images are generated. In step S615, a composite mesh is generated based on the plurality of meshes. In step S620, a left-eye 3D image and depth map are generated based on the composite mesh. In step S625, a right-eye 3D image and depth map are generated based on the composite mesh. In an exemplary embedding, the left-eye 3D image and depth map and the right-eye 3D image and depth map have a perspective field of view based on a receiver of the streaming 3D video. In step S630, the left-eye 3D image and depth map and the right-eye 3D image and depth map are streamed as a streaming 3D video. Here, a single 2D image can represent a single frame. The term "frame" can be understood as a single image that, when played in sequence with other frames of the video, creates motion on the playback surface. One mesh of the plurality of meshes may be generated for each of the plurality of 2D images. The generating step S615 may refer to combining or fusing the plurality of 2D images into a 3D representation of the scene. The viewpoint field of view may be the field of view of a user streaming the 3D image on a 3D playback device.

[0040] Example 2 The method of Example 1, wherein the plurality of 2D images may have a different perspective field of view compared to a receiver-based perspective field of view of the frames of the streaming 3D video.

[0041] Example 3: The method of Example 1, wherein the generating the plurality of meshes may include downsampling the plurality of 2D images to generate a plurality of feature maps corresponding to one of the plurality of 2D images, and generating the plurality of meshes based on the plurality of feature maps.

[0042] Example 4 The method of Example 3, wherein generating the left-eye 3D image and depth map and generating the right-eye 3D image and depth map may include combining the plurality of feature maps as a feature layered mesh, upsampling the feature layered mesh as a layered mesh, and generating the left-eye 3D image and depth map and generating the right-eye 3D image and depth map based on the layered mesh.

[0043] Example 5: The method of Example 4, wherein the combining of the plurality of feature maps may include initializing the plurality of feature maps to have a planar geometry and projecting the plurality of feature maps to generate a planar swept volume (PSV).

[0044] Example 6: The method of Example 4, wherein the feature layered mesh may include a plurality of channels, a first subset of the plurality of channels may include abstract network features, and a second subset of the plurality of channels may include depth and density information.

[0045] Example 7 The method of example 4, wherein the combining of the plurality of feature maps can include generating a visibility component to identify occlusions and dependencies across layers.

[0046] Example 8: The method of Example 4, wherein the combining of the multiple feature maps may include projecting the feature layer mesh onto at least one of the multiple feature maps to determine how closely the feature layer mesh approximates at least one of the multiple 2D images. Alternatively or additionally, the combining of the multiple feature maps may include projecting the feature layer mesh onto at least one of the multiple feature maps and comparing the result with at least one of the multiple 2D images to determine an approximation of the feature layer mesh to at least one of the multiple 2D images. Alternatively or additionally, the combining of the multiple feature maps may include projecting the feature layer mesh onto at least one of the multiple feature maps and comparing the result with at least one of the multiple 2D images to determine whether a difference between the feature layer mesh and at least one of the multiple 2D images satisfies a criterion. The criterion may include a pixel-wise delta threshold, a pixel-area average delta threshold, an object pixel delta threshold, a total loss threshold, a peak signal-to-noise ratio (PSNR), etc.

[0047] Example 9 The method of Example 1 may further include streaming the right-eye 3D image and depth map and the left-eye 3D image and depth map as the frames of the streaming 3D video.

[0048] Example 10 The method may include any combination of one or more of Examples 1-9.

[0049] Example 11: A non-transitory computer-readable storage medium comprising instructions, the instructions stored on the non-transitory computer-readable storage medium and configured, when executed by at least one processor, to cause a computing system to perform the method of any of Examples 1-10.

[0050] Example 12: An apparatus comprising means for carrying out the method according to any one of Examples 1 to 10.

[0051] Example 13. An apparatus comprising at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code configured to, using the at least one processor, cause the apparatus to perform at least the method described in any one of Examples 1 to 10.

[0052] Density and Alpha In an exemplary implementation, a quantity called density can be used, which is related to alpha via alpha = (1.0 - jnp.exp(-density)). Density can be given by the fog rendering equation and can work in an intuitive way: dense layers are mostly opaque and less dense layers are mostly transparent. However, alpha is linearly related to transparency, while density is logarithmically related to transparency. CNNs can reason more easily about density compared to alpha, and using density can help simplify compositing and gradient calculations (e.g., gradient calculation 445), leading to more efficient and faster networks. Here is a simple network that calculates density using a series of 2D convolutions: class DensityModel(nn.Module): @nn.compact def __call__(self, x): x = jax.nn.elu(nn.Conv(features=32, kernel_size=(3,3))(x)) x = jax.nn.elu(nn.Conv(features=32, kernel_size=(3,3))(x)) return jax.nn.softplus(nn.Conv(features=1, kernel_size=(3,3))(x))

[0053] In the final layer of the network, you can use the jax.nn.softplus nonlinearity, which forces the density to be strictly positive. Therefore, when you calculate alpha = exp(-density), the value will be between [0.0,1.0]. You can then combine RGB and density and pass that as the output of the network. Functions to perform direct overcompositing with rgb_density, if needed, include: rgb, density = jax_utils.subdivide(rgb_density, (3, 1), axis=-1) overed_image = composite.density_over(rgb, density, premultiply=True) If you want the alpha instead of the density, you can convert the density like this: rgba = composite.rgb_density_to_rgba(rgb, density)

[0054] Depth normalized coordinate (DNC) space The differentiable renderer can be optimized to render tightly connected 3D mesh geometry. The acceleration structures used in this rendering code assume this geometry is associated with a defined view frustum and near and far clipping planes. The renderer natively supports the concept of layers of geometry within this frustum, and while each layer is ray traced separately, many functions assume that layers contain RGB-plus-alpha textures that are composited in a fixed back-to-front order to produce the final rendered image.

[0055] The MpiOptions class can be used to describe the rendering viewpoint of the ray tracing code. This class establishes the viewport field of view, near and far planes, number of depth layers, and texture resolution. By specifying only one of horz_fov_degress or vert_fov_degrees, the other can be calculated automatically based on the aspect ratio of the texture width and height. viewport_options = MpiOptions( height=180, width=320, num_layers = 8, near_depth = 0.8, far_depth=100.0, horz_fov_degrees=130, vert_fov_degrees=104)

[0056]

number

[0057] where $h$ and $w$ are the height and width of the texture in pixels, and $\alpha_x$ and $\alpha_y$ are the horizontal and vertical fields of view of the viewport. The input coordinates $(x_v,y_v,z_v)$ are * viewport_space * This space is defined such that $+y$ is along the central primary ray looking into the viewport, $+z$ points straight down, and $+x$ completes the right-hand coordinate system. The output coordinates are homogeneous and must be normalized. Thus, a ray passing through $(x_v,y_v,z_v)$ will intersect the texture at pixel coordinate $(*\frac{u}{w},\frac{v}{w})$. Through translation, scaling, and rotation, * Viewport space * An external set can be defined for the set of viewpoints, w_f_viewport, that transform from to the desired world coordinates $(x_w,y_w,z_w)$.

[0058]

number

[0059] The matrix on the left is a slightly expanded version of the viewport specificity matrix. The matrix on the right captures the geometry of the view projection, where a point $z_v$ is projected to a normalized displacement $d$. The relationship between the two is a function of $d_n$ and $d_f$, which are the $z$-depths of the near and far clipping planes. Construction of the completednc_f_viewport transformation (the product of these two matrices) is achieved by calling this function: dnc_f_mpi=transforms.dnc_f_mpi_matrix(viewport_options.intrinsics, viewport_options.near_depth, viewport_options.far_depth)

[0060] One thing to note about the dnc_f_viewport transform is that the dnc "depth" value represents the distance along a ray originating at the viewport center of the projection. * not present * Rather, they are simply related to the zl values ​​in the layer coordinate space.

[0061] Converting DNC depth to a vertex grid One aspect of DNC space is that the resulting depth values ​​$d$ are normalized so that a value of 0.0 represents the far clipping depth and 1.0 represents the near clipping depth. Values ​​in between are linearly interpolated as $1 / z$. That is, $d$ behaves like a normalized disparity value that smoothly interpolates across the depth range of the viewing frustum, appearing to the viewer to be equal increments as the image is projected and rendered near the projection center of the frustum.

[0062] Being able to construct the space in this way means that the neural network can output features activated with the jax.nn.tanh() function, which are scaled between 0.0 and 1.0 and can then be interpreted as DNC depth values ​​$d$. This provides the network with a very natural and well-behaved way of describing depths that scale linearly when projected onto a target viewpoint (or viewpoints). Of course, in order to render the geometry described by a series of DNC depth layers, it is useful to be able to convert it into a conventional triangle mesh. To do this, vertex_grid_utils.vertex_grid_from_dnc_depths() can be used. The example below also shows how to composite the dnc_f_viewport matrix with a w_f_viewport transformation so that the resulting triangle mesh appears in world coordinates rather than viewport coordinates. # Define a viewport to world transform. Use the identity for now, but this could # be any coordinate transform related viewport space to world space. w_f_viewport = jnp.eye(4) w_f_dnc = jnp.matmul(w_f_viewport, jnp.linalg.inv(dnc_f_viewport)) # Contruct equally spaced DNC "depth" values ​​and then broadcast from a vector of # [D] depths to [D, H, W, 1] tensor containing DNC "layers" like those that # might be produced in a tanh() activated feature from a network. dnc_depths = jnp.linspace( start=0.0, stop=1.0, num=viewport_options.num_layers, endpoint=True, dtype=jnp.float32) dnc_depths = jnp.broadcast_to( jnp.expand_dims(dnc_depths, (1, 2, 3)), (viewport_options.num_layers, viewport_options.height, viewport_options.width, 1)) # Convert the dnc depths to an xyz triangle mesh. vertex_grid_w = vertex_grid_utils.vertex_grid_from_dnc_depths(w_f_dnc, dnc_depths) print(f'dnc_depths shape: {dnc_depths.shape}') print(f'vertex_grid_w shape: {vertex_grid_w.shape}')

[0063] In this example, we can construct a series of plane layers that are evenly distributed in DNC depth space. This results in a set of planes distributed at 1 / z units equal to world space, similar to how MPI planes can be constructed. Note also that the final transformation converts a single DNC depth value into a 3D (x_w,y_w,z_w) coordinate. While the abscissas in DNC space are implied by the pixel locations in two and three dimensions in the dnc_depth tensor, these abscissas are made explicit in vertex_grid_w. Another thing implied by both dnc_depths and vertex_grid_w is the connectivity of the mesh. Both describe a tightly connected mesh, using groups of four adjacent coordinates to form two triangles within the mesh. The term vertex grid can be used to describe a mesh whose topology is implied by the shape and structure of a tensor containing only the (x_w,y_w,z_w) of its vertices.

[0064] Exemplary embodiments may include a non-transitory computer-readable storage medium including instructions stored thereon, which, when executed by at least one processor, are configured to cause a computing system to perform any of the methods described above. Exemplary embodiments may include an apparatus including means for performing any of the methods described above. Exemplary embodiments may include an apparatus including at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to cause, using the at least one processor, to perform at least any of the methods described above.

[0065] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, which may be specialized or general-purpose, coupled to receive data and instructions from the storage system, and to transmit data and instructions to the storage system.

[0066] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0067] To provide user interaction, the systems and techniques described herein are implemented on a computer that has a display device (such as an LED (light-emitting diode), or OLED (organic LED), or LCD (liquid crystal display) monitor screen) for displaying information to the user, as well as a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide user interaction. For example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be acoustic, verbal, or tactile input.

[0068] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or that includes middleware components (e.g., an application server), or that includes front-end components (e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the systems and techniques described herein), or a combination of such back-ware, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0069] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and by virtue of the client-server relationship they have to each other.

[0070] Although a number of embodiments have been described, it will nevertheless be understood that various modifications may be made without departing from the spirit and scope of the present disclosure.

[0071] Additionally, the logic flows depicted in the figures do not require the particular order or sequential order shown to achieve desired results. Furthermore, other steps may be provided in or eliminated from the described flows, and other components may be added to or removed from the described systems. Accordingly, other embodiments are within the scope of the following claims.

[0072] As described herein, while certain features of the described embodiments have been illustrated, numerous modifications, substitutions, changes, and equivalents will occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and variations that fall within the scope of the embodiments. They are presented by way of example only, not limitation, and it is to be understood that various changes in form and detail may be made. Any portion of the apparatus and / or methods described herein may be combined in any combination except mutually exclusive combinations. The embodiments described herein may include various combinations and / or subcombinations of the functions, components, and / or features of the different embodiments described.

[0073] While exemplary embodiments may include various modifications and alternative forms, embodiments thereof have been shown by way of example in the drawings and will be described in detail herein. It should be understood, however, that the intention is not to limit the exemplary embodiments to any particular form, but rather to cover all modifications, equivalents, and alternatives falling within the scope of the claims. Like numerals refer to like components throughout the description of the figures.

[0074] Some of the above exemplary embodiments are described as a process or method that is depicted as a flowchart. While the flowcharts describe operations as sequential, many of the operations may occur in parallel, simultaneously, or concurrently. The order of operations may also be rearranged. A process may be terminated when its operations are completed, or may have additional steps not included in the drawings. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0075] The above methods, some of which are illustrated by flowcharts, may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine- or computer-readable medium such as a storage medium. A processor(s) may perform the necessary tasks.

[0076] Specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments, however, example embodiments may be embodied in many alternate forms and should not be construed as limited to only the embodiments set forth herein.

[0077] It is also understood that, while the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a "first element" may be referred to as a "second element," and similarly, a "second element" may be referred to as a "first element," without departing from the scope of the example embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0078] When an element is referred to as being "connected" or "coupled" to another element, it is understood that the element may be directly connected or directly coupled to another element, or there may be intervening elements. In contrast, when an element is referred to as being "directly connected" or "directly coupled" to another element, there are no intervening elements. Other words used to indicate the relationship between elements should be interpreted similarly (e.g., between versus directly between, adjacent versus directly adjacent, etc.).

[0079] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes," and / or "including," when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0080] It should also be noted that in some alternative implementations, the functions / acts shown may occur out of the order noted in the figures. For example, two figures shown in succession may, in fact, be executed concurrently or may sometimes be executed in the reverse order, depending upon the functions / acts involved.

[0081] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the exemplary embodiments belong. Furthermore, it will be understood that terms (e.g., as defined in commonly used dictionaries) should be interpreted to have a meaning consistent with their meaning in the context of the relevant art, and not in an idealized or overly formal sense unless expressly defined herein.

[0082] The exemplary embodiments above and corresponding portions of this Detailed Description are presented in terms of software, or algorithms and symbolic representations of operations on data bits within a computer memory. These descriptions and representations are the ones by which those skilled in the art effectively convey the substance of their work to others skilled in the art. An algorithm, as the term is used herein, and as it is used generally, is conceived to be a self-consistent sequence of steps leading to a desired result. These steps require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of optical, electrical, or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It proves convenient at times, principally for reasons of common usage, to refer to these symbols as "bits," "values," "elements," "symbols," "characters," "terms," ​​"numbers," or the like.

[0083] In the above exemplary embodiments, references to symbolic representations (e.g., in the form of flowcharts) of acts and operations that may be implemented as program modules or functional processes include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types, and may be written and / or implemented using existing hardware in existing structural elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits, field programmable gate array (FPGA) computers, etc.

[0084] It should be noted, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specified, or as will be apparent from the discussion, terms such as processing or calculating or computing or determining or displaying refer to the actions and processes of a computer system or similar electronic computing device that manipulate and convert data that are represented as physical, electronic quantities in the computer system's registers and memory into other data that are also represented as physical quantities in the computer system's memory or registers, or other such information storage, transmission, or display device.

[0085] It should also be noted that the software implementations of the exemplary embodiments are typically encoded on some form of non-transitory program storage medium or implemented via some type of transmission medium. The program storage medium may be magnetic (e.g., a floppy disk or hard drive) or optical (e.g., a compact disk read-only memory, or CD-ROM), and may be read-only or random-access. Similarly, the transmission medium may be twisted wire pairs, coaxial cable, optical fiber, or some other suitable transmission medium known in the art. The exemplary embodiments are not limited by these aspects of any given implementation.

[0086] Finally, it should also be noted that while the appended claims set forth particular combinations of features described herein, the scope of the disclosure is broadened to encompass any combination of features or embodiments disclosed herein, and is not limited to the particular combinations claimed below, regardless of whether that particular combination is specifically recited in the appended claims at this time.

Claims

1. 1. A method comprising: receiving a plurality of two-dimensional (2D) images representing frames of a streaming three-dimensional (3D) video; generating a plurality of meshes corresponding to the plurality of 2D images; generating a composite mesh based on the plurality of meshes; generating a left-eye 3D image and depth based on the synthetic mesh; generating a right-eye 3D image and depth map based on the synthetic mesh, wherein the left-eye 3D image and depth map and the right-eye 3D image and depth map have a viewpoint perspective based on a receiver of the streaming 3D video.

2. The method of claim 1 , wherein the plurality of 2D images have a different perspective field of view compared to a receiver-based perspective field of view of the frames of the streaming 3D video.

3. The generation of the plurality of meshes comprises: downsampling the plurality of 2D images to generate a plurality of feature maps corresponding to one of the plurality of 2D images; generating the plurality of meshes based on the plurality of feature maps; 3. The method of claim 1 or claim 2, comprising:

4. The generating of the left eye 3D image and depth map and the generating of the right eye 3D image and depth map comprises: combining the plurality of feature maps into a feature layered mesh; upsampling the feature layered mesh as a layered mesh; generating the left-eye 3D image and the depth map based on the layered mesh and generating the right-eye 3D image and the depth map; The method of claim 3, comprising:

5. 5. The method of claim 4, wherein the combining of the plurality of feature maps comprises initializing the plurality of feature maps to have a planar geometry and projecting the plurality of feature maps to generate a planar swept volume (PSV).

6. the feature layered mesh includes a plurality of channels; a first subset of the plurality of channels comprising abstract network characteristics; The method of claim 4 or claim 5, wherein a second subset of the plurality of channels includes depth and density information.

7. The method of claim 4 to 6, wherein the combining of the plurality of feature maps includes generating a visibility component to identify occlusions and dependencies across layers.

8. 8. The method of claim 4, wherein the combining of the plurality of feature maps comprises projecting the feature layer mesh onto at least one of the plurality of feature maps to determine how closely the feature layer mesh approximates at least one of the plurality of 2D images.

9. The method of claim 1 , further comprising streaming the right-eye 3D image and depth map and the left-eye 3D image and depth map as the frames of the streaming 3D video.

10. 10. A non-transitory computer-readable storage medium comprising instructions, the instructions being stored on the non-transitory computer-readable storage medium and configured to cause a computing system to perform a method according to any one of claims 1 to 9 when executed by at least one processor.

11. Apparatus comprising means for carrying out the method according to any one of claims 1 to 9.

12. 1. An apparatus comprising: at least one processor; at least one memory containing computer program code; Equipped with Apparatus, wherein said at least one memory and said computer program code are configured, by said at least one processor, to cause said apparatus to at least perform a method according to any of claims 1 to 9.

Images