Real-time view compositing

Abstract

1. A method comprising: receiving a plurality of two-dimensional (2D) images representing frames of a streaming three-dimensional (3D) video; generating a plurality of meshes corresponding to one of the plurality of 2D images; generating a composite mesh based on the plurality of meshes; generating a left-eye 3D image and depth map based on the composite mesh; and generating a right-eye 3D image and depth map based on the composite mesh, wherein the left-eye 3D image and depth map and the right-eye 3D image and depth map have a viewpoint perspective based on the receiver of the streaming 3D video.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of U.S. Provisional Application No. 63 / 383,866, filed Nov. 15, 2022, the disclosure of which is incorporated herein by reference in its entirety.

[0002] Embodiments relate to rendering three - dimensional left - eye and right - eye images.

Background Art

[0003] Generating a three - dimensional (3D) image from a plurality of two - dimensional (2D) images may involve stitching the 2D images. The stitching operation can include simultaneously calculating all possible translations (x, y, z) between two 2D images for a 3D view. By calculating the translations, the best overlap regarding the cross - correlation measure can be determined. When three or more input images are used, the exact placement of parts of the images (sometimes called tiles) can be globally optimized (e.g., the resulting 3D image is corrected to remove gaps and overlaps).

Summary of the Invention

[0004] Exemplary embodiments describe a neural rendering and view synthesis system configured to synthesize two viewpoints (e.g., left - eye and right - eye) based on the position of a viewer's eyes on the receiver side of a streaming sequence of 3D images. The 3D images are synthesized as a hierarchical mesh based on a plurality of 2D images and can be rendered before streaming a sequence of 3D images. Using the hierarchical mesh, any potential viewpoint perspective of a user viewing the 3D image using a playback device can be rendered.

[0005] Exemplary embodiments will be better understood from the detailed description below and the accompanying drawings of this specification. Similar elements are indicated by similar reference numerals and are given for illustrative purposes only and are not limiting to the exemplary embodiments. [Brief explanation of the drawing]

[0006] [Figure 1] A block diagram of a streaming pipeline according to an exemplary embodiment is shown. [Figure 2] A block diagram of a view compositing system according to an exemplary embodiment is shown. [Figure 3] A block diagram of a neural rendering system according to an exemplary embodiment is shown. [Figure 4A] A block diagram of an exemplary machine learning downsampling network according to an exemplary embodiment is shown. [Figure 4B] A block diagram of an exemplary machine learning view synthesis network according to an exemplary embodiment is shown. [Figure 4C] A block diagram of an exemplary machine learning upsampling network according to an exemplary embodiment is shown. [Figure 5] A block diagram of the system according to an exemplary embodiment is shown. [Figure 6] A block diagram of the method according to an exemplary embodiment is shown. [Modes for carrying out the invention]

[0007] It should be noted that these figures are intended to illustrate general characteristics of the methods and / or structures used in specific exemplary embodiments and to supplement the written descriptions provided below. However, these drawings may not be to scale and may not accurately reflect the exact structural or performance characteristics of any given embodiment, and should not be construed as defining or limiting the range of values ​​or characteristics encompassed by the exemplary embodiments. For example, the arrangement of modules and / or structural elements may be reduced or exaggerated for clarity. The use of similar or identical reference numbers in various drawings is intended to indicate the existence of similar or identical elements or features.

[0008] Generating 3D images by stitching together multiple 2D images is not fast enough for real-time streaming of 3D images. In other words, capturing and stitching 2D images in real-time 3D streaming applications is too slow to provide the desired user experience. Existing solutions may reduce the resolution (e.g., number of pixels) to very low resolutions, and even then, they may not achieve the frame rate or frames per second (fps) desired for real-time streaming of 3D images.

[0009] An exemplary embodiment can use a machine learning model trained to synthesize a 3D mesh from multiple 2D images. The exemplary embodiment can further use the trained machine learning model to render the synthesized 3D mesh to generate two 3D images, each having a viewpoint field of view (e.g., left eye and right eye). The 3D images can then be streamed to a 3D playback device. In a real-time 3D streaming application, the exemplary embodiment can stream 3D images at a sufficiently high resolution (e.g., 4K) and frame rate (e.g., 30fps) to provide a desired user experience.

[0010] Figure 1 shows a block diagram of a 3D streaming pipeline according to an exemplary embodiment. As shown in Figure 1, the 3D streaming pipeline 110 may include a mesh synthesis module 115 and a rendering module 120. The 3D streaming pipeline 110 can be configured to receive multiple 2D images 105 and generate two 3D images that can be streamed to a playback device 125.

[0011] Multiple cameras (e.g., a camera rig) can be configured to capture multiple 2D images 105. In an exemplary embodiment, the multiple cameras (e.g., six cameras) may be time-synchronized rolling shutter RGB cameras sharing exposure and white balance settings. The multiple 2D images 105 can represent image frames. In an exemplary embodiment, the multiple cameras may not capture depth. Therefore, the multiple 2D images 105 may not contain depth information. In an exemplary embodiment, the 3D streaming pipeline 110 may reside on the same device as the multiple cameras (e.g., a transmission station). In this exemplary embodiment, once a frame(s) is captured, the multiple 2D images 105 can be processed inline by the 3D streaming pipeline 110.

[0012] In an exemplary embodiment, the 3D streaming pipeline 110 may reside on different devices (e.g., a server) from the multiple cameras. In this exemplary embodiment, once a frame(s) is captured, the multiple 2D images 105 may be compressed using the HEVC (h.265) standard at, for example, 25 Mbps per camera, and then communicated to the server for processing by the 3D streaming pipeline 110. In this embodiment, the multiple 2D images 105 may be decompressed and processed by the 3D streaming pipeline 110.

[0013] The mesh synthesis module 115 can be configured to synthesize or merge multiple 2D images 105 into a 3D representation of the scene, sometimes called a layered mesh (e.g., a layered mesh 20, which will be described in more detail below), and the rendering module 120 can be configured to render the layered mesh as a 3D image representation of the scene. In an exemplary embodiment, the layered mesh can represent a complete 3D scene based on multiple 2D images 105. In other words, the layered mesh is not configured to render any particular viewpoint field of view (e.g., left eye and right eye) of a user viewing the playback device 125. Furthermore, the layered mesh can be used to render any potential viewpoint field of view of a user viewing the playback device 125. In other words, the composite layered mesh can represent the field of view of any view corresponding to any head position, and so the left and right eye images rendered based on the composite layered mesh when displayed on the playback device 125 can have a field of view that can be modified in 6 degrees of freedom (DoF) based on the user view and / or head position.

[0014] The rendering module 120 can be configured to render two images and generate two depth maps (e.g., one for each eye of the user viewing the playback device 125). The two images may be an RGB and a depth map view. In an exemplary embodiment, the playback device 125 can communicate the current or latest viewpoint field of view and / or head pose of the user viewing the playback device 125. Thus, the rendering module 120 can be configured to render two images and generate two depth maps (e.g., an RGB and a depth map view) based on the user's current or latest viewpoint field of view and / or head pose. The rendered images and generated depth maps can be streamed (e.g., communicated) to the playback device 125 at, for example, 4K resolution and 30fps.

[0015] In an exemplary embodiment, the playback device 125 may be configured to perform a last-second reprojection of the rendered image and generated depth map using the most recent user-estimated viewpoint field of view and / or head pose before rendering to the playback device 125's display. The reprojection can adjust for user movement (e.g., changes in viewpoint field of view and / or head pose) during the streaming process (e.g., due to system and / or streaming latency).

[0016] Figure 2 shows a block diagram of a view synthesis system according to an exemplary embodiment. The view synthesis system can be configured to blend the weights and densities of images and reconstruct the depth layers in the form of a hierarchical mesh representation. The view synthesis system can be based on the DeepView algorithm. For example, the view synthesis system can be configured to generate a hierarchical mesh output through an iterative refinement process. As shown in Figure 2, the mesh synthesis module 115 includes a displacement correction module 205, a downsampling module 210, a synthesis module 215, and an upsampling module 220.

[0017] The displacement correction module 205 can be configured to generate displacement-corrected images 5. In an exemplary embodiment, the displacement correction module 205 can be configured to reproject each of the images 105 onto a hierarchical mesh plane as a displacement-corrected image 5. In an exemplary embodiment, the hierarchical mesh plane may be a proximal clipping plane. The displacement correction module 205 can be configured to reduce the resolution of each of the images 105 during each reprojection of the images 105. Each reprojection of the images 105 can ensure, or can help ensure, that the image coordinates are consistent across each of the displacement-corrected images 5.

[0018] The downsampling module 210 can be configured, for example, to downsample each of the deviation-corrected images 5 by 8x using a trained machine learning downsampling network. The machine learning downsampling network can be configured to generate a low-resolution feature map 10 for each of the deviation-corrected images 5.

[0019] FIG. 4A shows a block diagram of an exemplary machine learning downsampling network as an exemplary element (or embodiment) of the downsampling module 210. The machine learning downsampling network 405 can include a plurality of convolutional layers 410-1, 410-2, 410-3, 410-4, 410-5, 410-6, 410-7. For example, the machine learning downsampling network 405 can include a series of strided convolutional layers. Thus, the convolutional layers 410-1, 410-2, 410-3, 410-4, 410-5, 410-6, 410-7 can be strided convolutional layers. By spanning across the convolutional layers, the number of pixels by which the filter matrix of the convolutional layer moves across the entire input image is shown. The stride length of the convolutional layer indicates the number of steps executed when sliding the filter matrix across the entire image. In some embodiments, the stride length of the convolutional layers 410-1, 410-2, 410-3, 410-4, 410-5, 410-6, 410-7 can be 1 or 2. For example, the convolutional layers 410-1, 410-3, 410-5, and 410-7 can have a stride length of 1, and the convolutional layers 410-2, 410-4, and 410-6 can have a stride length of 2.

[0020] The plurality of convolutional layers 410-1, 410-2, 410-3, 410-4, 410-5, 410-6, 410-7 can be configured to reduce the resolution of the offset-corrected image 5. For example, it can be configured to reduce the resolution of the offset-corrected image 5 by 8 times (8x). For example, the convolutional layers 410-3, 410-4 can be configured to reduce the resolution of the offset-corrected image 5 by 2 times (2x), the convolutional layers 410-5, 410-6 can be configured to reduce the resolution of the offset-corrected image 5 by 2 times (2x), and the convolutional layer 410-7 can be configured to reduce the resolution of the offset-corrected image 5 by 2 times (2x), resulting in a total resolution reduction of 8 times (8x). In some embodiments, the convolutional layers 410-1, 410-2, 410-3, 410-4, 410-5, 410-6, 410-7 can be configured to increase the respective channel count of the offset-corrected image 5 to, for example, 4 to 32 channels.

[0021] Referring back to FIG. 2, the synthesis module 215 can be configured to generate the feature hierarchical mesh 15. The feature hierarchical mesh 15 can be a low-resolution hierarchical mesh. In an exemplary embodiment, the feature hierarchical mesh 15 can include 288×184 pixels×16 layers. FIG. 4B shows a block diagram of an exemplary machine learning view synthesis network according to an exemplary embodiment as an exemplary element (or embodiment) of the synthesis module 215.

[0022] As shown in Figure 4B, in an exemplary embodiment, the layers of the feature-hierarchical mesh 15 can be initialized to have a flat geometry, and the feature map 10 can be projected onto these layers to generate a planar sweep volume (PSV). Next, according to the exemplary embodiment, the initialization network (e.g., image-to-layer transition 415-1 and neural network 420-1 (e.g., convolutional neural network (CNN))) can be configured to compute initial estimates of the feature-hierarchical mesh 15 (e.g., neural network 420-1 or the output of the hierarchical mesh) based on the PSV. At this point, the layers of the feature-hierarchical mesh 15 can include network features (having 32 channels). In the exemplary embodiment, the first 30 channels of the machine learning view synthesis network can include abstract network features (e.g., since the features do not have a particular physical interpretation, the network is free to use the features of the feature-hierarchical mesh 15 in any useful way). However, in the exemplary embodiment, the last two channels of the feature-hierarchical mesh 15 can be used by the machine learning view synthesis network to derive depth and density information. Therefore, a machine learning view synthesis network can be configured to generate (or learn to generate) features for learning depth and density information.

[0023] The feature-hierarchical mesh 15 generated by the initialization network can be refined through two consecutive update steps. The first update step (update 1) may include a layer-to-image transition 425-1, an image-to-layer transition 415-2, a neural network 420-2 (e.g., CNN), a visibility component 430-1, and an activation block 435-1. The second update step (update 2) may include a layer-to-image transition 425-2, an image-to-layer transition 415-3, a neural network 420-2 (e.g., CNN), a visibility component 430-2, and an activation block 435-2. During each update step, the current feature-hierarchical mesh 15 can be backprojected into the input feature map 10. In some embodiments, the current feature-hierarchical mesh 15 can be compared to determine how well it approximates the actual image captured by the input camera. However, the feature-hierarchical mesh 15 backprojected into the input feature map 10 can be used in any useful process. In an exemplary embodiment, when projecting features of the feature-hierarchical mesh 15 onto the viewpoint of the input feature map 10, the geometry derived from the second-to-last depth channel of the hierarchical mesh can be used. This channel may be activated with tanh nonlinearity (e.g., activation blocks 435-1, 435-2, 435-3), scaled by the layer width, and added to a set of depth anchors (e.g., the output of activation blocks 435-1, 435-2, 435-3) that are equally spaced by parallax. By constructing the geometry using this technique, each layer can have its own parallax band, thus preventing layer overlap of the feature-hierarchical mesh 15.

[0024] Next, using the layer geometry, we can warp from layer space to view space (and back again). While in view space, we can perform composite operations using the latest channel (e.g., density features) of the feature layering mesh 15 to help communicate visibility information across layers. These visibility components 430-1, 430-2, and 430-3 can be used or assist in updating the network because they can infer occlusion and understand the dependencies across layers. The visibility components 430-1, 430-2, and 430-3 can be accumulated over net transparency and may include net transparency. The accumulated may include a reconstruction of the scene from behind a plane, and net transparency may be a soft occlusion mask of the plane.

[0025] The computation of visibility components 430-1, 430-2, and 430-3 can improve the functionality of the machine learning view synthesis network because the computation of visibility components can be a time for information to be communicated between layers. To complete the update step, visibility components 430-1, 430-2, and 430-3 can be warped back from each of the input view spaces to a central hierarchical mesh representation, and then these features can be input into the update network. The update network can be configured to generate differences that can be added via residual connections to the hierarchical mesh computed in the previous iteration. This allows the feature hierarchical mesh 15 to be generated iteratively (e.g., over multiple update steps) based on the feature map 10. Furthermore, the activation block 435-3, the layers 425-3 to the image, and the visibility component 430-3 can together generate a gradient computation 445. Alternatively (or additionally), layers 425-3 to the image can compute visibility components 430-3 (as gradient calculations 445) based on the depth calculated by the feature layering mesh 15 and activation blocks 435-3. This reconstruction, checking, and then refinement strategy implemented in the update step can be repeated several times, and the strategy can function like an iterative optimization algorithm. Convergence to a high-quality solution may occur in several iterations (e.g., 3 times).

[0026] Referring back to Figure 2, the upsampling module 220 can be configured to generate a hierarchical mesh 20. In an exemplary embodiment, the upsampling module 220 can be configured to increase the resolution of the feature hierarchical mesh 15. In an exemplary embodiment, the upsampling module 220 can be configured to increase the density 30 at resolution by, for example, eight times (8x). For example, the density 30 of the feature hierarchical mesh 15 can be increased to a resolution of 1080p. In an exemplary embodiment, the upsampling module 220 can be configured to refine the blend weights 25 of the feature hierarchical mesh 15. However, the blend weights 25 and mesh vertices 35 of the hierarchical mesh 20 may remain at a low resolution. Efficiency can be increased by leaving the blend weights 25 and mesh vertices 35 at a low resolution. For example, the final 3D image may not be affected by the resolution of the blend weights 25 and mesh geometry. In contrast, the final 3D image may be affected by the alpha and RGB resolutions.

[0027] Figure 4C shows a block diagram of an exemplary machine learning upsampling network by an exemplary embodiment as an exemplary element (or embodiment) of the upsampling module 220. As shown in Figure 4C, in the exemplary embodiment, the machine learning upsampling network 480 can use the feature-layered mesh 15 computed by the view synthesis network 440 and the final set of visibility components (gradient calculation 445), and then process these using a series of convolutions 450-1, 450-2, 450-3, 450-4, 450-5, 450-6, 450-7, 450-8, 450-9, 450-10, and 450-11, concatenations 455-1, 455-2, and a squeeze and excitation network (including a softmax 465 with features and weights, multiplicative and additive elements) to estimate the low-resolution blend weights 25, the vertex positions 35 of the mesh, and the higher-resolution density 30 layers of the layered mesh 20. In some embodiments, convolutions 450-7, 450-8 can be called blend models, and convolutions 450-9, 450-10, 450-11 can be called density models. In exemplary embodiments, the above density 30 layers can be upsampled via depth2space transform 470. In some embodiments, the feature hierarchical mesh 15 (e.g., the second to last channel of the feature hierarchical mesh 15) can be activated and transformed (475) to generate a tensor containing the vertex positions 35 of the 3D mesh.

[0028] Figure 3 shows a block diagram of a neural rendering system according to an exemplary embodiment. As shown in Figure 3, the rendering module 120 can be configured to generate a left-eye (LE) image 25 and a right-eye (RE) image 30 based on a layered mesh 30. In the exemplary embodiment, the LE image 25 and the RE image 30 can each be a 4k RGB plus depth image. As shown in Figure 3, the rendering module 120 can include an RE projection module 305, an LE projection module 310, an RE blend module 315, an LE blend module 320, an RE overcompositing module 325, and an LE overcompositing module 330. In the exemplary embodiment, the layered mesh 20 can include 16 mesh layers, each mesh layer including associated blend weights and density values. Furthermore, the layered mesh 20 can be used to render any potential viewpoint field of view of a user viewing the playback device 125. In other words, the composite layered mesh can represent the field of view of any view corresponding to any head position, and so the left and right eye images rendered based on the composite layered mesh when displayed on the playback device 125 can have a field of view that can be modified in 6 degrees of freedom (DoF) based on the user view and / or head position.

[0029] In an exemplary embodiment, each mesh layer of the hierarchical mesh 20 can be rasterized into the image space of the rendered output view. This process returns the intersection triangle index and centroid coordinates of each pixel in the output view. In the exemplary embodiment, this information can be used to (1) find a set of six blend weights (e.g., using centroid interpolation of the blend weights of the mesh layers generated by the model), (2) find density values ​​(e.g., using centroid interpolation of the mesh layer densities generated by the model), and (3) calculate the 3D coordinates of the intersections. These 3D coordinates can be back-projected onto the image plane of the original high-resolution input view to determine the RGB values ​​from each input image (e.g., using bilinear interpolation).

[0030] In an exemplary embodiment, blend weights may be activated using softmax nonlinearity and then used to calculate a simple weighted average of RGB values ​​from the input view. Density values ​​may be activated via softplus nonlinearity and converted to alpha values. In an exemplary embodiment, RGB plus alpha values ​​may be generated for each pixel in a layer by rasterizing one layer to the output view.

[0031] This sequence of rasterization, projection, and sampling steps includes the signal flow shown in Figure 3. For example, the signal flow includes projecting the input image onto mesh geometry (RE projection module 305 and LE projection module 310), blending the mesh together using blend weights, combining with alpha values ​​from the layers, and then projecting the result onto the output view (RE blend module 315 and LE blend module 320).

[0032] In exemplary embodiments, nonlinear activation may include post-resampling activation. In other words, nonlinear activation may include projecting the input image, blend weights, and density onto the output viewpoint. As the network is trained through a differentiable version of the rendering process described above, the network can learn to expect these activations to occur in higher resolution spaces, and the network can indeed learn to leverage post-interpolation activation to produce sharper, higher-resolution images even when working with relatively low-resolution blend weights and alpha density. In exemplary embodiments, post-resampling activation may be referred to as lazy rendering or lazy sampling of a layered mesh. Lazy rendering or lazy sampling of a layered mesh may help achieve high-quality 4K output even when the layered mesh contains 1080p density and low-resolution (e.g., 135 x 240 pixels) geometry and blend weights.

[0033] In an exemplary embodiment, rasterization can be repeated for each of the 16 mesh layers of the hierarchical mesh 20, and the resulting RGB plus alpha layer can then be rendered using alpha compositing to produce a final RGB image for a specific output viewpoint (RE overcompositing module 325 and LE overcompositing module 330). In an exemplary embodiment, the RE overcompositing module 325 and LE overcompositing module 330 can be configured to generate a depth channel by replacing the RGB of each layer with the parallax of that layer, and then compositing the parallax plus alpha layer.

[0034] In exemplary embodiments, delayed rendering or delayed sampling techniques have the advantage of separating the resolution of the input view, network output, and the final RGB plus depth rendering result. This is advantageous because the resolution of each component can be adjusted individually, which is useful in the trade-off between performance and quality. For example, the speed of an RGB plus depth renderer can be increased by outputting an RGB plus depth image at 1440p while using a 4k input.

[0035] The layered mesh 20 described above can be used to represent the 3D structure of a scene. In addition, the layered mesh provides a mesh representation that can be rasterized to generate an RGB plus depth view used for streaming to the playback device 125. The layered mesh may be related to multi-planar images (MPI). Using both MPI and layered meshes, the viewpoint can be rendered using a combination of field projection and alpha compositing. The mesh layers can occupy parallax bands that can be spaced equally in parallax (1 / z) space, similar to MPI planes. The view can be rendered by first projecting the layers onto the output viewpoint and then alpha compositing from back to front. However, unlike the planar geometry of MPI, the mesh layers may have network-generated geometry that molds itself to the shape of the object being reconstructed. This allows layered meshes to achieve similar quality to MPI with far fewer layers. Layered meshes enable a more efficient (10 times faster than using MPI) method of performing learned upsampling and rendering, so 30fps at high resolution can be achieved using layered meshes.

[0036] Figure 5 shows a block diagram of a system according to an exemplary embodiment. In the example of Figure 5, the system (e.g., wearable device 300, augmented reality system, virtual reality system, companion device, etc.) may include a computing system or at least one computing device and should be understood to represent virtually any computing device configured to implement the technologies described herein. Thus, the device may be understood to include various components that may be used to implement the technologies described herein, or different or future versions thereof. As an example, the system may include a processor 505 and memory 510 (e.g., non-temporary computer-readable memory). The processor 505 and memory 510 may be coupled (e.g., communicatively coupled) by a bus 515.

[0037] The processor 505 may be used to execute instructions stored in at least one memory 510. Thus, the processor 505 may implement various features and functions described herein, or additional or alternative features and functions. The processor 505 and at least one memory 510 may be used for various other purposes. For example, at least one memory 510 may represent examples of various types of memory and associated hardware and software that may be used to implement any one of the modules described herein.

[0038] At least one memory 510 may be configured to store data and / or information related to the device. At least one memory 510 may be a shared resource. Thus, at least one memory 510 may be configured to store data and / or information related to other elements in a larger system (e.g., image / video processing or wired / wireless communication). The processor 505 and at least one memory 510 can be used together to implement the techniques described herein. Thus, the techniques described herein can be implemented as code segments (e.g., software) stored on the memory 510 and executed by the processor 505. Thus, the memory 510 can include any combination of the mesh synthesis module 115 and the rendering module 120. The exemplary embodiment shown in Figure 5 is merely one example of a hardware configuration. In other embodiments, operations can be shared among computing devices.

[0039] Example 1 Figure 6 shows a block diagram of the method according to an exemplary embodiment. As shown in Figure 6, in step S605, multiple two-dimensional (2D) images representing frames of a streaming three-dimensional (3D) video are received. In step S610, multiple meshes corresponding to the multiple 2D images are generated. In step S615, a composite mesh is generated based on the multiple meshes. In step S620, a left-eye 3D image and depth are generated based on the composite mesh. In step S625, a right-eye 3D image and depth map are generated based on the composite mesh. In the exemplary embedding, the left-eye 3D image and depth map, as well as the right-eye 3D image and depth map, have a viewpoint field of view based on the receiver of the streaming 3D video. In step S630, the left-eye 3D image and depth map, as well as the right-eye 3D image and depth map, are streamed as a streaming 3D video. Here, a single 2D image can represent a single frame. The term “frame” can be understood as a single image that, when played sequentially with other frames of the video, creates motion on the playback plane. One of multiple meshes may be generated for each of the multiple 2D images. The generation step S615 may refer to compositing or merging the multiple 2D images into a 3D representation of the scene. The viewpoint field of view may be the field of view of the user streaming the 3D images on a 3D playback device.

[0040] Example 2 The method according to Example 1, wherein the plurality of 2D images may have different viewpoint fields compared to the viewpoint field based on the receiver of the frames of the streaming 3D video.

[0041] Example 3 The method according to Example 1, wherein the generation of the plurality of meshes may include downsampling the plurality of 2D images to generate a plurality of feature maps corresponding to one of the plurality of 2D images, and generating the plurality of meshes based on the plurality of feature maps.

[0042] Example 4 The method according to Example 3, wherein the generation of the left eye 3D image and depth map, and the generation of the right eye 3D image and depth map, may include synthesizing the plurality of feature maps as a feature-hierarchical mesh, upsampling the feature-hierarchical mesh as a hierarchical mesh, and generating the left eye 3D image and depth map and the right eye 3D image and depth map based on the hierarchical mesh.

[0043] Example 5 The method according to Example 4, wherein the synthesis of the plurality of feature maps may include initializing the plurality of feature maps to have a flat geometry and projecting the plurality of feature maps to generate a planar sweep volume (PSV).

[0044] Example 6: The method according to Example 4, wherein the feature-hierarchical mesh may include a plurality of channels, a first subset of the plurality of channels may include abstract network features, and a second subset of the plurality of channels may include depth and density information.

[0045] Example 7 The method according to Example 4, wherein the synthesis of the plurality of feature maps may include generating visibility components to identify occlusion and layer-wide dependencies.

[0046] Example 8 The method according to Example 4, wherein the synthesis of the plurality of feature maps may include projecting the feature-hierarchical mesh onto at least one of the plurality of feature maps in order to determine how well the feature-hierarchical mesh approximates at least one of the plurality of 2D images. Alternatively or additionally, the synthesis of the plurality of feature maps may include projecting the feature-hierarchical mesh onto at least one of the plurality of feature maps and comparing the result with at least one of the plurality of 2D images in order to determine how well the feature-hierarchical mesh approximates at least one of the plurality of 2D images. Alternatively or additionally, the synthesis of the plurality of feature maps may include projecting the feature-hierarchical mesh onto at least one of the plurality of feature maps and comparing the result with at least one of the plurality of 2D images in order to determine whether the difference between the feature-hierarchical mesh and at least one of the plurality of 2D images satisfies a criterion. The criterion may include a per-pixel delta threshold, a region-averaged delta threshold for pixels, an object pixel delta threshold, a total loss threshold, and a peak signal-to-noise ratio (PSNR), etc.

[0047] Example 9 The method of Example 1 may further include streaming the right eye 3D image and depth map, and the left eye 3D image and depth map, as frames of the streaming 3D video.

[0048] Example 10 The method may include one or more combinations of Examples 1 to 9.

[0049] Example 11 A non-temporary computer-readable storage medium containing instructions, wherein the instructions are stored in the non-temporary computer-readable storage medium and, when executed by at least one processor, are configured to cause a computing system to perform the method described in any of Examples 1 to 10.

[0050] Example 12: An apparatus comprising means for performing the method described in any of Examples 1 to 10.

[0051] Example 13: An apparatus comprising at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code are configured to cause the apparatus to perform at least one of the methods described in any of Examples 1 to 10 using the at least one processor.

[0052] Density and alpha In exemplary embodiments, a quantity called density can be used, which is related to alpha via alpha = (1.0 - jnp.exp(-density)). Density can be given by the fog rendering equation and can function in an intuitive way, with high-density layers being mostly opaque and low-density layers being mostly transparent. However, while alpha has a linear relationship to transparency, density has a logarithmic relationship to transparency. CNNs can infer more easily about density compared to alpha, and the use of density can help simplify composite and gradient calculations (e.g., gradient calculation 445), which leads to more efficient and faster networks. Here is a simple network that calculates density using a series of 2D convolutions. class DensityModel(nn.Module): @nn.compact def __call__(self, x): x = jax.nn.elu(nn.Conv(features=32, kernel_size=(3,3))(x)) x = jax.nn.elu(nn.Conv(features=32, kernel_size=(3,3))(x)) return jax.nn.softplus(nn.Conv(features=1, kernel_size=(3,3))(x))

[0053] In the final layer of the network, we can use the jax.nn.softplus nonlinearity to make the density strictly positive. Therefore, when we calculate alpha=exp(-density), the value will fall between [0.0, 1.0]. We can then combine the RGB and density and pass them as the network output. If necessary, a function for performing rgb_density and direct overcompositing can include the following: rgb, density = jax_utils.subdivide(rgb_density, (3, 1), axis=-1) overed_image = composite.density_over(rgb, density, premultiply=True) If you need alpha instead of density, you can convert the density as follows: rgba = composite.rgb_density_to_rgba(rgb, density)

[0054] Depth normalized coordinate (DNC) space A differentiable renderer can be optimized to render tightly connected 3D mesh geometry. The acceleration structure used in this rendering code assumes that this geometry is associated with a defined frustum and proximal and distal clipping planes. The renderer natively supports the concept of layers of geometry within this frustum, with each layer being ray-traced separately, while many functions generate the final rendered image assuming that the layers contain RGB plus alpha textures, which are composited in a fixed back-to-front order.

[0055] The MpiOptions class can be used to describe the rendering viewpoint for ray tracing code. This class establishes the viewport field of view, proximal and distal planes, number of depth layers, and texture resolution. By specifying only one of horz_fov_degress or vert_fov_degrees, the other can be automatically calculated based on the aspect ratio of the texture width and height. viewport_options = MpiOptions( height=180, width=320, num_layers = 8, near_depth = 0.8, far_depth=100.0, horz_fov_degrees=130, (vert_fov_degrees=104)

[0056]

number

[0057] Here, $h$ and $w$ are the height and width of the texture in pixels, and $\alpha_x$ and $\alpha_y$ are the horizontal and vertical field of view of the viewport. The input coordinates $(x_v, y_v, z_v)$ are: * Viewport Space * Located within this space, $+y$ is aligned with the central principal ray that the viewport examines, $+z$ points straight down, and $+x$ completes the coordinate system on the right. The output coordinates are homogeneous and need to be normalized. Thus, a ray passing through $(x_v, y_v, z_v)$ intersects the texture at pixel coordinates $(*\frac{u}{w},\frac{v}{w})$ via translation, scaling, and rotation. * Viewport space * An external set can be defined for the set of viewpoints, w_f_viewport, that are converted from the desired world coordinates $(x_w, y_w, z_w)$.

[0058]

number

[0059] The matrix on the left is a slightly extended version of the viewport specificity matrix. The matrix on the right captures the geometry of the view projection, where point $z_v$ is projected onto a normalized displacement $d$. The relationship between the two is a function of $d_n$ and $d_f$, which are the $z$ depths of the proximal and distal clipping planes. The construction of the completdnc_f_viewport transformation (the product of these two matrices) is achieved by calling this function. dnc_f_mpi=transforms.dnc_f_mpi_matrix(viewport_options.intrinsics, viewport_options.near_depth, (viewport_options.far_depth)

[0060] One thing to note about the dnc_f_viewport transformation is that the dnc "depth" value represents the distance along the rays that originate at the center of the projection viewport. * not present * Rather, they are simply associated with the $z_l$ value in sheath coordinate space.

[0061] Converting DNC depth to vertex grid One aspect of the DNC space is that the resulting depth value $d$ is normalized such that a value of 0.0 represents the distal clipping depth and 1.0 represents the proximal clipping depth. Values ​​in between are linearly interpolated by $1 / z$. In other words, $d$ behaves like a normalized disparity value that smoothly interpolates across the depth range of the viewing frustum, so that to the viewer, the image appears to be a single increment when projected and rendered near the projection center of the frustum.

[0062] The ability to construct space in this way means that the neural network can output features activated by the jax.nn.tanh() function, which scale between 0.0 and 1.0 and can then be interpreted as DNC depth values ​​$d$. This provides the network with a very natural and well-behaved way of describing depth that scales linearly when projected onto a target viewpoint (or multiple viewpoints). Of course, it is useful to be able to convert the geometry described by a series of DNC depth layers into a conventional triangular mesh for rendering. To do this, vertex_grid_utils.vertex_grid_from_dnc_depths() can be used. The example below also shows how to combine the dnc_f_viewport matrix with the w_f_viewport transformation so that the resulting triangular mesh appears in world coordinates rather than viewport coordinates. # Define a viewport to world transform. Use the identity for now, but this could # be any coordinate transform related viewport space to world space. w_f_viewport = jnp.eye(4) w_f_dnc = jnp.matmul(w_f_viewport, jnp.linalg.inv(dnc_f_viewport)) # Contruct equally spaced DNC "depth" values ​​and then broadcast from a vector of # [D] depths to [D, H, W, 1] tensor containing DNC "layers" like those that # might be produced in a tanh() activated feature from a network. dnc_depths = jnp.linspace( start=0.0, stop=1.0, num=viewport_options.num_layers, endpoint=True, dtype=jnp.float32) dnc_depths = jnp.broadcast_to( jnp.expand_dims(dnc_depths, (1, 2, 3)), (viewport_options.num_layers, viewport_options.height, viewport_options.width, 1)) # Convert the dnc depths to an xyz triangle mesh. vertex_grid_w = vertex_grid_utils.vertex_grid_from_dnc_depths(w_f_dnc, dnc_depths) print(f'dnc_depths shape: {dnc_depths.shape}') print(f'vertex_grid_w shape: {vertex_grid_w.shape}')

[0063] In this example, we can construct a series of plane layers evenly distributed in DNC depth space. This results in a set of planes distributed in $1 / z$ units equal to world space, similar to how MPI planes can be constructed. Note that the final transformation also converts a single DNC depth value to 3D $(x_w,y_w,z_w)$ coordinates. The ax-coordinates in DNC space are implied by the pixel positions in two and three dimensions of the dnc_depth tensor, but these ax-coordinates are explicitly shown in vertex_grid_w. Another thing implied by both dnc_depths and vertex_grid_w is the connectivity of the mesh. Both represent closely connected meshes that form two triangles within the mesh using groups of four adjacent coordinates. The term vertex grid can be used to describe a mesh whose topology is implied by the shape and structure of a tensor containing only the $(x_w,y_w,z_w)$ of its vertices.

[0064] An exemplary embodiment may include a non-temporary computer-readable storage medium containing stored instructions, which, when executed by at least one processor, cause a computing system to perform the method described in any of the methods described above. An exemplary embodiment may include a device that includes means for performing any of the methods described above. An exemplary embodiment may include a device that includes at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code are configured to cause at least one processor to perform at least one of the methods described above.

[0065] Various embodiments of the systems and technologies described herein may be implemented in digital electronic circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include embodiments in one or more computer programs executable and / or interpretable on a programmable system which may be special or general-purpose, comprising at least one programmable processor, at least one input device, and at least one output device, coupled to receive data and instructions from and transmit data and instructions to a storage system.

[0066] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, apparatus and / or device (e.g., magnetic disks, optical disks, memory, programmable logic circuits (PLDs)) used to provide machine instructions and / or data to a programmable processor that contains a machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0067] To provide user interaction, the systems and technologies described herein are implemented on a computer having a display device (LED (light-emitting diode), OLED (organic LED), or LCD (liquid crystal display) monitor screen) for displaying information to the user, as well as a keyboard and pointing device (e.g., mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide user interaction. For example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and the input from the user can be acoustic, spoken language, or tactile input.

[0068] The systems and technologies described herein may be implemented in computing systems that include backend components (e.g., as data servers), middleware components (e.g., application servers), or frontend components (e.g., client computers having a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or combinations of such backend, middleware, or frontend components. The components of the system may be interconnected by digital data communications (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), and the Internet.

[0069] A computing system can include clients and servers. Clients and servers are generally geographically distant from each other and typically interact through a communication network. The client-server relationship arises from computer programs running on each computer and from the client-server relationship they have with each other.

[0070] Numerous embodiments have been described. Nevertheless, it is understood that various modifications can be made without departing from the spirit and scope of this specification.

[0071] Additionally, the logic flow shown in the diagram does not require a specific sequence, i.e., a sequential order, to achieve the desired result. Furthermore, other steps may be added to the described flow, or steps may be removed from the described flow, and other components may be added to the described system, or other components may be removed from the described system. Accordingly, other embodiments are within the scope of the following claims.

[0072] While specific features of the embodiments described herein have been illustrated, numerous modifications, substitutions, alterations, and equivalents will be conceivable to those skilled in the art. It should be understood that the appended claims are intended to encompass all such modifications and alterations that fall within the scope of the embodiments. They are presented only as examples, not as limitations, and various modifications in form and detail are permitted. Any part of the apparatus and / or method described herein may be combined in any combination, except for mutually exclusive combinations. The embodiments described herein may include various combinations and / or partial combinations of the functions, components, and / or features of the different embodiments described herein.

[0073] The exemplary embodiments may include a variety of modifications and alternative forms, which are shown as examples in the drawings and described in detail herein. However, it should be understood that the exemplary embodiments are not intended to limit themselves to any particular form, but rather are intended to cover all modifications, equivalents, and alternatives that fall within the scope of the claims. Similar figures refer to similar components throughout the description of the drawings.

[0074] Some of the exemplary embodiments described above are explained as processes or methods shown as flowcharts. While flowcharts describe operations as sequential, many operations may be performed in parallel, simultaneously, or concurrently. The order of operations can also be changed. A process may terminate when its operations are completed, but it may have additional steps not shown in the diagrams. A process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0075] Some of these are illustrated by flowcharts. The methods described above can be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segment for performing the required task may be stored in a machine- or computer-readable medium such as a storage medium. A processor(s) may perform the required task.

[0076] Certain structural and functional details disclosed herein are representative only for illustrative purposes to illustrate exemplary embodiments. However, exemplary embodiments are embodied in many alternative forms and should not be construed as being limited only to the embodiments described herein.

[0077] Furthermore, while terms such as “first,” “second,” etc., may be used herein to describe various elements, it should be understood that these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, “first element” may be referred to as “second element” without departing from the scope of the exemplary embodiments, and similarly, “second element” may be referred to as “first element.” As used herein, the term “and / or” includes one or any combination of the related items described.

[0078] When an element is referred to as being "connected" or "joined" to another element, it should be understood that the element can be directly connected to or directly joined to another element, or that an intermediary element may exist. In contrast, when an element is referred to as being "directly connected" or "directly joined" to another element, no intermediary element exists. Other words used to indicate relationships between elements should be interpreted similarly (e.g., between to directly, adjacent to directly adjacent, etc.).

[0079] The terms used herein are for the sole purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. Where used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context otherwise explicitly indicates. It should be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” when used herein, specify the presence of the described features, integers, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof.

[0080] Furthermore, it should be noted that in some alternative embodiments, the functions / actions shown may occur in a different order than that shown in the diagrams. For example, two diagrams shown consecutively may actually be performed in reverse order, depending on the functions / actions that can be performed simultaneously or are involved.

[0081] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as those generally understood by those skilled in the art to which the exemplary embodiments belong. Furthermore, terms (for example, those defined in commonly used dictionaries) should be interpreted as having meanings consistent with their meanings in the context of the relevant art, and it will be understood that they should not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0082] The exemplary embodiments described above and the corresponding embodiments for carrying out the inventions are presented with respect to software, or to algorithms and symbolic representations of operations on data bits in computer memory. These descriptions and representations are intended to effectively convey the essence of the work to those skilled in the art. An algorithm is considered, where the term is used herein and in general usage, to be a self-consistent sequence of steps leading to a desired result. These steps require the physical manipulation of physical quantities. Usually, but not always necessary, these quantities take the form of optical, electrical, or magnetic signals that are stored, transferred, combined, compared, and otherwise manipulable. For reasons of general use, it is sometimes convenient to refer to these symbols as “bits,” “values,” “elements,” “symbols,” “characters,” “terms,” “numbers,” etc.

[0083] In the exemplary embodiments described above, references to symbolic representations (e.g., in the form of flowcharts) of actions and behaviors that may be implemented as program modules or functional processes include routines, programs, objects, components, data structures, etc., which perform specific tasks or implement specific abstract data types and may be described and / or implemented using existing hardware in existing structural elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate array (FPGA) computers, etc.

[0084] However, it should be noted that all these and similar terms are associated with appropriate physical quantities and are merely convenient labels applied to those quantities. Unless otherwise specified, or as is evident from the discussion, terms such as processing, calculation, computation, or display determination refer to the actions and processes of a computer system or similar electronic computing device that manipulate and convert data, which is represented as physical and electronic quantities within the registers and memory of the computer system, into other data. This other data, in turn, is represented as a physical quantity within the memory or registers of the computer system, or any other such information storage, transmission, or display device.

[0085] It should also be noted that the software embodiments of the exemplary embodiments are typically encoded on some form of non-temporary program storage medium or implemented via some type of transmission medium. The program storage medium may be magnetic (e.g., floppy disk or hard drive) or optical (e.g., compact disk read-only memory, i.e., CD-ROM), and may be read-only or random-access. Similarly, the transmission medium may be twisted wire pairs, coaxial cables, optical fibers, or any other suitable transmission medium known in the art. The exemplary embodiments are not limited to these aspects of any given embodiment.

[0086] Finally, while the appended claims describe specific combinations of the features described herein, it should be noted that the scope of this disclosure is extended to encompass any combination of the features or embodiments disclosed herein, and is not limited to any specific combination claimed below, regardless of whether such combination is specifically enumerated in the appended claims at this point.

Claims

1. It is a method, Receiving multiple two-dimensional (2D) images representing frames of streaming three-dimensional (3D) video, To generate multiple meshes corresponding to the aforementioned multiple 2D images, A composite mesh is generated based on the aforementioned multiple meshes, Based on the aforementioned composite mesh, a 3D image of the left eye and a depth map are generated. The process includes generating a right-eye 3D image and depth map based on the composite mesh, wherein the left-eye 3D image and depth map and the right-eye 3D image and depth map have a viewpoint perspective based on the receiver of the streaming 3D video. The generation of the aforementioned plurality of meshes is To generate multiple feature maps corresponding to one of the aforementioned multiple 2D images, The process of generating the multiple meshes based on the multiple feature maps, Methods that include...

2. The method according to claim 1, wherein the plurality of 2D images have different viewpoints compared to the viewpoints based on the receiver of the frames of the streaming 3D video.

3. The method according to claim 1 or 2, wherein the generation of the plurality of feature maps includes downsampling the plurality of 2D images in order to generate a plurality of feature maps corresponding to one of the plurality of 2D images.

4. A method, Receiving multiple two-dimensional (2D) images representing frames of streaming three-dimensional (3D) video, To generate multiple meshes corresponding to the aforementioned multiple 2D images, A composite mesh is generated based on the aforementioned multiple meshes, Based on the aforementioned composite mesh, a 3D image of the left eye and a depth map are generated. The process includes generating a right-eye 3D image and depth map based on the composite mesh, wherein the left-eye 3D image and depth map and the right-eye 3D image and depth map have a viewpoint perspective based on the receiver of the streaming 3D video. The generation of the aforementioned plurality of meshes is To generate multiple feature maps corresponding to one of the multiple 2D images, the multiple 2D images are downsampled, The process of generating the multiple meshes based on the multiple feature maps, Includes, The generation of the left eye 3D image and depth map, and the generation of the right eye 3D image and depth map, The above-mentioned multiple feature maps are combined as a feature-hierarchical mesh, Upsampling the aforementioned feature-hierarchical mesh as a hierarchical mesh, Based on the aforementioned hierarchical mesh, the left eye 3D image and the depth map are generated, and the right eye 3D image and the depth map are generated. Methods that include...

5. The method according to claim 4, wherein the synthesis of the plurality of feature maps comprises initializing the plurality of feature maps to have a flat geometry and projecting the plurality of feature maps to generate a planar sweep volume (PSV).

6. The aforementioned feature-hierarchical mesh includes multiple channels, The first subset of the plurality of channels includes abstract network features, The method according to claim 4, wherein a second subset of the plurality of channels includes depth and density information.

7. The method according to claim 4, wherein the synthesis of the plurality of feature maps includes generating visibility components to identify occlusion and layer-wide dependencies.

8. The method according to claim 4, wherein the synthesis of the plurality of feature maps includes projecting the feature-hierarchical mesh onto at least one of the plurality of feature maps in order to determine how well the feature-hierarchical mesh approximates at least one of the plurality of 2D images.

9. The method according to any one of claims 1, 2, and 4, further comprising streaming the right eye 3D image and depth map, and the left eye 3D image and depth map as frames of the streaming 3D video.

10. A computer program including instructions, wherein, when executed by at least one processor, the instructions are configured to cause a computing system to perform the method according to any one of claims 1, 2, and 4.

11. An apparatus comprising means for performing the method according to any one of claims 1, 2, and 4.

12. It is a device, At least one processor, At least one memory containing computer program code, Equipped with, The apparatus is configured such that the at least one memory and the computer program code are configured by the at least one processor to cause the apparatus to perform at least one of the methods according to claim 1, claim 2, and claim 4.

Images