Stereoscopic rendering using raymarching and a broadcaster for a virtual view of such rendering.

Abstract

Methods for providing stereoscopic images, comprising: Generating a monoscopic set of rendered images; and Converting the set of rendered images into a stereoscopic image pair using depth information from the monoscopic set of rendered images and raymarching, characterized in that the monoscopic set of rendered images is a cube image (310) with both a color buffer and a depth buffer and the stereoscopic image pair is a stereoscopic pair of cube images.

Description

TECHNICAL AREA

[0001] This application is generally directed towards rendering and in particular towards the rendering of stereoscopic images. BACKGROUND

[0002] Many computer graphics are created by mathematically modeling the interaction of light with a three-dimensional (3D) scene from a specific viewpoint. This process, known as "rendering," produces a two-dimensional (2D) image of the scene from that viewpoint and is analogous to photographing a real scene.

[0003] With the increasing demand for computer graphics, and especially real-time computer graphics, computer systems with graphics processing subsystems to accelerate the rendering process have become widespread. In these systems, the rendering process is divided between a computer's general-purpose central processing unit (CPU) and the graphics processing subsystem, which is architecturally centered around a graphics processing unit (GPU). Typically, the CPU performs higher-level operations, such as determining the position, movement, and collision of objects within a given scene. From these higher-level operations, the CPU generates a set of rendering instructions and data that define the desired rendered image(s). For example, rendering instructions and data might define scene geometry, lighting, shading, texturing, motion, and / or camera parameters for a scene.The graphics processing subsystem creates one or more rendered images from the set of rendering commands and data.

[0004] With the accelerated rendering provided by these computer systems, various applications, such as video games, utilize cloud-based rendering. This technology allows users to enjoy a virtual graphics experience on their computer devices, including virtual reality head-mounted displays (HMDs). The cloud-based platform also enables users to compete remotely against others in tournaments or esports events via a communication network. In addition to those actively participating in a video game or application, passive spectators—those watching but not playing or actively participating—can view an ongoing game on their computer device from the perspective of one of the players at video game tournaments or other esports events.US 2016 / 0 353 090 A1 describes a method for converting images from a planar image plane to a spherical image plane. US 2016 / 0 219267 A1 describes a method according to the preamble of claim 1. SHORT DESCRIPTION

[0005] In one aspect, the invention provides a method for providing stereoscopic images according to claim 1. In one embodiment, the method includes, among other things: (1) generating a monoscopic set of rendered images and (2) converting the set of rendered images into a stereoscopic image pair using depth information from the monoscopic set of rendered images and raymarching.

[0006] In another aspect, the invention provides a broadcaster for a virtual view according to claim 11. In one embodiment, the broadcaster for a virtual view includes, among other things: (1) a cloud-based renderer configured to generate a monoscopic set of rendered 360-degree images and to use raymarching and depth information to convert the set of rendered images into a stereoscopic 360-degree image pair, and (2) an image processor configured to encode the stereoscopic image pair for video transmission.

[0007] In a further aspect, the invention provides a cloud-based renderer according to claim 16. In one embodiment, the cloud-based renderer includes (1) a memory and (2) at least one processor coupled to the memory and configured to convert a cube image into a stereoscopic pair of cube images using raymarching and a depth buffer of the cube image. SHORT DESCRIPTION

[0008] The following description, in conjunction with the accompanying drawings, refers to the following: Fig. 1. A block diagram of an embodiment of a broadcasting system for a virtual view, constructed in accordance with the principles of the invention, illustrates; Fig. 2 a block diagram of an embodiment of a virtual view broadcasting system constructed in accordance with the principles of the invention, illustrating the operational sequence for providing a virtual view stream of a stereoscopic image pair to a client computing device; Fig. 3A, Fig. 3B, Fig. 3C and Fig. 3D diagrams illustrate the application of raymarching to a cube image according to the principles of the invention; Fig. Figure 4 illustrates an embodiment of a method for providing a stereoscopic panorama from a monoscopic set of rendered images, carried out in accordance with the principles of the invention; Fig. 5 a diagram of an example of an omnidirectional stereo (ODS) projection used in accordance with the principles of the invention in converting a monoscopic set of rendered images into a stereoscopic image pair; Fig. 6. A diagram illustrates an example of using geometry to determine a viewing direction with respect to a cube image in accordance with the principles of the invention; Fig. Figure 7 illustrates a diagram of an example of distortion removal from a stereoscopic cube image in accordance with the principles of the invention; and Fig. Figure 8 illustrates a diagram of an example of the application of translucency in stereo in accordance with the principles of the invention. DETAILED DESCRIPTION

[0009] With the recent growth in virtual reality headsets, 360-degree panoramic photos and videos have become increasingly popular. There are two types of such panoramas: monoscopic and stereoscopic. Monoscopic panoramas are generally easy to capture, process, and display, but they obviously lack any depth information. Stereoscopic panoramas are much more complex to capture or generate.

[0010] One way to capture surrounding depth in just two images is to project the objects onto the images such that each column of image pixels corresponds to a set of rays emanating from a point on a projection circle whose diameter is equal to the interpupillary distance (IPD). This can be compared to a normal perspective projection, where all rays originate from the same point: the center of the camera. A projection circle is a circle with a diameter equal to the IPD that surrounds a projection center used to calculate the regular perspective projection. The projection circle corresponds to eye positions when one turns their head, with the projection center being the pivot point.

[0011] When creating stereo panoramas from photographs or rendered images, the number of images captured along the horizontal direction should be maximized, and each image should be taken from a different point on the aforementioned circle. The central vertical bands of these images can then be stitched and blended to approximate the intended specific projection. For an ideal result, the number of input images for each eye should equal the width of the output panorama in pixels—which is, of course, impractical. However, using fewer images produces a panorama that, when viewed from any direction other than perpendicular to any pair of source images (right and left), will exhibit inaccurate depth representation.Any number of source images suitable for real-time rendering obviously produce incorrect depth in most viewing directions, accompanied by significant stitching artifacts on nearby objects, visible as ghosting.

[0012] It is recognized here that rendering techniques not bound to a regular perspective projection—such as ray tracing or raymarching with analytical scene rendering—can be used to generate a projection without stitching artifacts, including the circular projection required for stereo panoramas. Using the disclosed rendering processes, a viewpoint can be shifted by tracing rays emanating from points on a projection circle. The original rendered image (non-circular) from the original perspective is used as source data for tracing rays originating from the points on the projection circle.

[0013] The invention provides a (round) transmitter or broadcaster for a virtual view or virtual view broadcaster that uses a set of rendered images, including their depth buffers, to calculate a regular perspective projection originating from a point near the projection center of the original environment image or images, but not necessarily coinciding with the exact center. The set of rendered images can be a monoscopic set of 360-degree rendered images, such as a monoscopic cube image. Similarly, two projections, one for each eye, can then be calculated for each position and orientation of a user, such as from the head of a user wearing a head-mounted display (HMD).

[0014] The disclosed broadcaster for a virtual view computes a stereoscopic panorama with a perfect or near-perfect circular projection using an algorithm that utilizes the monoscopic set of rendered images and their depth buffers. A renderer can use scene geometry, a scene description, and so on to represent the monoscopic set of images. The monoscopic set of rendered images can be a cube image with six square images, each representing a perspective projection of a view along one of the principal axes. Each of the sides, or faces, of the cube image contains depth information. For any given ray, surface information can be determined by finding the intersection of the ray with the surfaces described by the depth buffers of the cube image.

[0015] In one embodiment, raymarching is used to emit each ray corresponding to an output pixel in the panorama from its own pixel point, i.e., the viewpoint of each pixel. In some embodiments, raymarching can be performed using a depth view image. In some embodiments, ten taps of a depth mipmap are sufficient to determine the surface to be used. A depth mipmap, also called a MIP image, consists of pre-computed, optimized sequences of images, each sequence of images being a progressively lower resolution of the same image. In embodiments that use a depth buffer, mipmaps are generated by finding the nearest surface among the higher-resolution input pixels, which are used to compute a single lower-resolution output pixel.This differs from color mipmaps, where color pixels are typically simply averaged. Fig. 3A to 3D are an example of the use of mipmaps during raymarching.

[0016] One difference between calculating a regular perspective projection and a stereoscopic panorama is that the desired ray origin and direction are calculated for each output pixel. For calculating a regular perspective projection, the origin is the projection center (eye position), and the direction can be calculated using view and projection matrices and pixel coordinates. For calculating a stereoscopic panorama, the direction can be calculated based on the output pixel coordinates within the panorama, and the ray origin is calculated based on the panorama center and direction, and lies on the projection circle. Consequently, when calculating panoramic views, regardless of whether the projection or the layout of monoscopically rendered views is a cube or an equirect, the ray origin can be determined using the same method.The disclosed method, system, or device determines a viewing direction for each output pixel, corresponding to that pixel. For example, an equirectangular or equiangular projection for x, y can be used to determine the viewing direction by defining a pixel position geometry.

[0017] For stereoscopic panoramas, a projection point lies on the projection circle. Since the projection circle lies in the horizontal plane, a projection point is located where the projection of a direction vector onto the horizontal plane is tangent to the projection circle. For a given direction, the projection point can be adjusted to lie on the projection circle. There are two projection points on the projection circle, separated by the diameter. One of the projection points is for the left eye and one for the right eye. With respect to a cube image, geometry can be used to determine a viewing direction. The following is discussed. Fig. Section 6 provides an example.

[0018] Once the ray origin and direction are determined, raymarching is used. That is, the ray is projected onto the environment image(s), and the depth buffers are sampled along the ray until an intersection point is found between the ray and the implicit surface encoded by the depth buffer.

[0019] In some embodiments, the ray origin may not coincide with one of the projection centers of the images, and there may be insufficient information about the scene due to parallax and occlusion. Simply put, if you take a picture of a tree, you don't know what is directly behind that tree—but this information may be important from other viewpoints. To compensate, the disclosed broadcaster is configured for a virtual view, or part thereof, such that when the basis is a set of images, these images can be captured from slightly different viewpoints, significantly increasing the likelihood that every required surface is found in at least one image. If a cube image is used, a second layer of that cube image can be rendered.This multi-layered rendering is known as depth peeling and can provide additional data for occlusion processing. The raymarching algorithm can then select the correct layer, for example, based on the similarity between the current point on the ray and both depth values ​​in the cube image, or other information such as surface orientation or depth gradients.

[0020] Now returning to the characters, illustrates Fig. Figure 1 shows a block diagram of an embodiment of a broadcasting system for a virtual view 100, constructed in accordance with the principles of the invention. The broadcasting system for a virtual view 100 comprises a virtual view broadcaster 110, computer devices 120, 130, 140, 150, 160, and a communication network 170. The computer devices 120, 130, 140, 150, 160 are collectively referred to as computer devices 120-160 and are communicatively connected to the virtual view broadcaster 110 via the communication network 170. The communication network 170 can be a conventional network, such as the Internet, which enables connected computer devices to communicate with each other.

[0021] The Virtual View Broadcaster 110 is configured to generate streams of virtual views of a stereoscopic image pair from a monoscopic set of rendered images. The monoscopic set of rendered images can be generated by a virtual camera positioned within an application. In one embodiment, specific code can be provided to game publishers or other application developers, which they can integrate into their game or application to install virtual cameras. The provided code can match a renderer of the Virtual View Broadcaster 110 to provide the rendered images when a specific virtual camera is selected. The Virtual View Broadcaster 110 can receive a selection input from a camera selector to determine the virtual camera to use.In some embodiments, multiple selection inputs for different virtual cameras can be received, and the broadcaster for a virtual view 100 can simultaneously generate the virtual views for the selected multiple virtual cameras. The camera selector can be a specific person or a computing device belonging to that person, which is used to provide the selection input to the broadcaster for a virtual view 100. The computing device can be one of the computing devices 120-160.

[0022] The Virtual View Broadcaster 110 can be implemented on one or more servers of a cloud computing platform. The Virtual View Broadcaster 100 provides a set of monoscopically rendered images, which are then used to generate a stereoscopic image pair for 360-degree virtual views. Accordingly, the Virtual View Broadcaster 100 can create a virtual view experience from within an application that is not a virtual reality application.

[0023] The broadcaster for a virtual view 110 can include at least one CPU and multiple GPUs. In the illustrated embodiment, the broadcaster for a virtual view 110 includes a renderer 111, an image processor 112, a video transmitter 113, a memory 114, and an application engine 115. The memory 114 can be conventional memory or memory typically used for servers. The application engine 115 contains the operating instructions corresponding to the scene-generating algorithms, such as a game engine that delivers scenes from a video game.

[0024] Renderer 111 generates a set of monoscopically rendered images and converts them into a stereographic image pair using raymarching. Renderer 111 can be a cloud-based renderer that generates the set of monoscopically rendered images from various virtual cameras positioned within an application. In one embodiment, the rendered images correspond to six faces of a cube image for a virtual camera location. Renderer 111 can generate other rendered images that can be combined to provide a 360-degree virtual view. In some embodiments, the rendered images can be used for equilateral views instead of cube images. The rendered images correspond to application data received by, for example, the computing devices 120-160 of the virtual viewers. The application data can include scene data.In some embodiments, the application data can include directional information that specifies a viewing direction in which a virtual viewer looks, selects, points, etc.

[0025] As mentioned above, Renderer 111 can provide the set of monoscopically rendered images by rendering six images that form a cube at the camera location. The cube image is then used to texture a scene in the application. Cube image texturing is a form of texture mapping that uses a 3D direction vector to index into a texture consisting of six square 2D textures arranged like the faces of a cube. Because rendering cube images is more complex than rendering normal views into 2D textures, GPUs are often used in conjunction with a CPU. For example, Nvidia Grid™ technology can be used to provide the renderers and rendering procedures disclosed herein to support image rendering, such as cube image rendering.Nvidia Grid™, from Nvidia Corporation of Santa Clara, California, provides a graphics virtualization platform that enables the use of Nvidia GPU technology for virtual desktops and applications. The Renderer 111 can include additional components such as memory and a communication interface.

[0026] The image processor 112 encodes the stereoscopic image pair into streams of virtual views for transmission. The image processor 112 can also provide additional functions such as reformatting and image processing. The encoded stereoscopic image pair is then provided to the video transmitter 113 and sent to the computing devices 120-160. The video transmitter 113 can be a conventional device that receives encoded individual frames and transmits them as a video stream. Instead of being contained within the virtual view broadcaster 110, in some embodiments the video transmitter 113 can be conventionally coupled with the virtual view broadcaster 100 and used to transmit the streams of virtual views of the stereoscopic image pair. The video transmitter 113 is a video proxy server.

[0027] The streams of virtual views of stereoscopic images provided to the various computing devices 120-160 may originate from the same virtual camera and may be the same video stream. In some embodiments, a virtual viewer can determine their actual view based on the directional information provided by their respective computing devices 120-160, such as an HMD.

[0028] The computing devices 120-160 can be VR headsets, smartphones, desktop computers, laptops, computer pads, tablets, etc. These devices can be thin clients that communicate with the broadcaster for a virtual view 110 and provide it with sufficient application data for rendering and stitching. Each or at least some of the computing devices 120-160 can be different types of devices. For example, the computing devices 120-140 can be VR headsets, the computing device 150 can be a laptop, and the computing device 160 can be an Nvidia SHIELD tablet.

[0029] Fig. Figure 2 illustrates a block diagram of an embodiment of a virtual view broadcasting system 200, which is constructed in accordance with the principles of the invention and illustrates the operational sequence for providing streams of virtual views of a stereoscopic image pair to a client computing device. The virtual view broadcasting system 200 includes a virtual view broadcaster 210, a video transmitter 220, and a client computing device 230. In some embodiments, the virtual view broadcaster 110 is Fig. 1 is configured as the broadcaster for a virtual view 210 and operates as such.

[0030] The virtual view broadcaster 210 is configured to generate a stream of a virtual view of a stereoscopic image pair for the client computing device 230. The stereoscopic image pair can originate from one or more virtual cameras positioned in an application based on application data. In one embodiment, the virtual view broadcaster 210 is a cloud-based device, such as a cloud server (or servers), that generates content in the cloud, which is streamed to the client computing device 230. The virtual view broadcaster 210 can reside on one or more network servers (e.g.,(Cloud servers) to render content in a real-time (or semi-real-time) content engine and create a set of monoscopically rendered images used to generate a 360-degree video for streaming a stereoscopic image pair to virtual viewers in real-time or semi-real-time. Real-time is a level of responsiveness that is perceived by a user as sufficiently immediate or that allows a processor to keep pace with the processing of an input. In some embodiments, the input data can be processed within milliseconds, so that it is available as feedback virtually instantly. The broadcaster for a virtual view 210 includes a renderer 211 and an image processor 218. In some embodiments, the broadcaster for a virtual view 210 also includes the video transmitter 220. The renderer 211 can be a cloud-based renderer.

[0031] Renderer 211 generates rendered datasets of monoscopic images and converts these monoscopically rendered images into a stereoscopic image pair. Renderer 211 uses image-based raymarching to convert the monoscopic set of rendered images with color and depth into the stereoscopic image pair. Raymarching can provide a perfect or near-perfect circular projection with minor artifacts due to disocclusion, which can be corrected with available artifact workarounds. After raymarching, Renderer 211 can perform additional rendering steps, such as applying translucent geometry processing and post-translucency effects (tone mapping) to each of the stereoscopic image pairs.

[0032] Renderer 211 can generate images for applications or technologies that require multiple real-time renderings, such as for VR or AR displays. The datasets can correspond to multiple views (e.g., cube views). Thus, Renderer 211 can draw six rendered images that can be used to provide 360-degree views. Using a game application as an example, Renderer 211 can generate the six rendered images by calling the game engine's rendering function six times.

[0033] The renderer 211 includes a central processing unit (CPU) 212 coupled with several graphics processing units (GPUs) 213, 215, 217. The number of CPUs or GPUs can vary in different renderer embodiments. The GPUs can be configured to render entire, separate panoramic views, or each GPU can render a frame or single image of the same view. In some embodiments, the broadcaster for a virtual view 210 can include multiple renderers or a single renderer with multiple CPUs, each coupled with multiple GPUs.

[0034] Renderer 211 includes a memory 214 containing a set of operating instructions that control the operation of Renderer 211. These operating instructions correspond to algorithms that align the rendering process in accordance with the stereoscopic rendering disclosed herein. Memory 214 can be coupled with CPU 212 and GPUs and cooperate with CPU 212 and GPUs 213, 215, and 217 to display the stereoscopic image pair. Renderer 211 can also include or interact with other components, such as an application engine, coupled with CPU 212, memory 214, or GPUs 213, 215, and 217, which are used to generate the rendered images.

[0035] CPU 212 is configured to work in conjunction with GPUs 213, 215, and 217 to generate the monoscopic set of rendered images. CPU 212 can send scene information to GPUs 213, 215, and 217 for further processing to generate images or frames for a specific viewpoint. For example, CPU 212 can use application data from client computing device 230 to determine scene information such as vertex positions, triangle connections, textures, material properties, light sources, and so on, and pass this information to GPUs 213, 215, and 217 to paint the pixels of the rendered images. In some embodiments, client computing device 230 can send action or motion information that indirectly affects the vertex positions over communication network 170. Therefore, the application data of the client computing device 230 can only contain a portion of the scene data.

[0036] Renderer 211 sends the generated stereoscopic image pair to image processor 218. The stereoscopic image pair can be a 360-degree stereoscopic image pair. Image processor 218 performs compression and encodes the image into a video stream for transmission, reformatting, and image processing. Image processor 218 may include an encoder that encodes into standard protocol video streams such as H.264, HEVC, etc. Image processor 218 then sends the encoded streams of virtual views of the stereoscopic image pair to video transmitter 220 for streaming.

[0037] The video transmitter 220 receives the encoded video stream and transmits the virtual view stream of the stereoscopic image pair to the client computing device 230. The video transmitter 220 can be a video proxy server. The client computing device 230 can be one of the computing devices 120-160 of Fig. Be 1.

[0038] The client computing device 230 displays a virtual view using the virtual video stream of the stereoscopic image pair received from the video transmitter 220. The client computing device can also send application data to the broadcaster for a virtual view 210, which is used for real-time content generation and processing.

[0039] Fig. 3A, Fig. 3B, Fig. 3C and Fig. 3D diagrams illustrate the application of raymarching to a cube image in accordance with the principles of the invention. A renderer as disclosed herein can perform raymarching to convert a monoscopic cube image into a stereoscopic cube image.

[0040] Fig. Figure 3A illustrates a two-dimensional view of a cube image 310. Within the cube image 310 is a projection circle 320 with a projection center 325. The cube image 310 has several faces, one of which is labeled 315. Several rays, generally referred to as rays 340, are shown radiating from several projection points on the projection circle 320 and passing over the frustum or truncated boundaries of the faces of the cube image 310. All of the rays 340 originate from the projection circle 320, but some of them pass only as far as one face of the cube image 310, while others intersect between several faces of the cube image 310. In a three-dimensional cube image, a single ray can pass through three different faces of the cube image.

[0041] Fig. Figure 3B illustrates an example of a cross-section of a frustum or truncated 360 of a cube image surface, such as surface 315. The frustum 360 has a number of image points or pixels in the horizontal dimension (three are shown in Figure 3B). Fig. 3B), each with its own stump. For each pixel, there is a depth value representing the distance to a surface. During raymarching, the renderer searches for the surface that first intersects the projected ray. The ray in Fig. 3B to 3D can be one of the rays 340 from Fig. It should be 3A.

[0042] Fig. 3C illustrates the 360 ​​stump, which has been transformed into a post-projection space to facilitate working with it: pixels correspond to boxes instead of generic stubs. In Fig. 3C is the depth buffer, a simple height map. The renderer can use the height map by comparing each surface along the ray to the ray itself.

[0043] Fig. 3D provides a more efficient use of depth information. In Fig. In 3D, two lower-resolution mipmap layers, 1 (red) and 2 (green), have been added. These additional mipmap layers allow the renderer to traverse the surfaces of layer 2 until it is hit by the beam. Then, the renderer switches to the surfaces of layer 1 until it is hit by the beam. Finally, the renderer switches to surfaces of layer 0 until the last hit is found.

[0044] Raymarching is typically performed with a view where a ray is projected and traversed until a surface is found. The revealed rendering process can traverse an entire cube image at once. The raymarching algorithm detects which face to start with, then performs raymarching on that face, and then on another face as the ray moves from one face to the other. The first face to start with might be the one where the ray origin lies within the truncated surfaces; and then the raymarching begins with that face. If the ray origin is near the center of the cube image, outside the truncated surfaces, then the first face to be raymarched is the one that intersects the ray first.In some embodiments, a simple raybox intersection algorithm can be used to find this surface and the intersection point because the nearby planes of the six truncated faces form a cube. Each ray originates at a specific location on a surface, and as soon as the ray encounters a wall, such as a wall separating surfaces, the raymarching process for that specific ray switches to that other surface. The raymarching algorithm has access to all six surfaces of a cube image and knows the geometry that connects them. Thus, the rays can be processed sequentially across a cube image, from one ray to the next. In some embodiments, the rays can be processed in parallel. Switching between the different surfaces can be done in a shader program for each individual ray.

[0045] Fig. Figure 4 illustrates an embodiment of Method 400 for providing a stereoscopic panorama from a monoscopic cube image. Method 400 can be performed by a broadcaster disclosed herein for a virtual view. In some embodiments, a renderer, such as a cloud-based renderer, can perform at least some of the steps of Method 400. For some of the steps, an application engine, such as a game engine, can be used. The steps of Method 400 correspond to an algorithm that generates stereoscopic images from monoscopic images, as disclosed herein. Method 400 begins in step 405 with the intention of rendering an opaque geometry into a cube image.

[0046] In step 410, the opaque geometry is rendered into a cube image. A conventional cube image rendering method can be used. In step 420, pretranslucency effects are applied to the rendered cube image. The pretranslucency effects can include screen space ambient occlusion (SSAO) and screen space reflections (SSR).

[0047] In step 430, a stereoscopic pair of cube images is generated from the rendered monoscopic cube image using raymarching and the depth information from the cube image. In one embodiment, the stereoscopic pair of cube images is extrapolated from the rendered monoscopic cube image. A renderer can convert the rendered monoscopic cube image into the stereoscopic cube image pair using raymarching. As in Fig. When a 3A to 3D representation is used, raymarching finds a surface represented by a depth buffer that intersects a projected ray. The raymarching algorithm can reside within an application engine. In some implementations, the raymarching functionality can be extracted as a library and deployed separately from an application engine.

[0048] In step 440, the translucency is rendered into both cube images of the stereoscopic image pair. In one embodiment, the translucency is rendered after raymarching using a vertex shader to match the circular projection. In these embodiments, geometric information for translucent objects is available. In the vertex shader, the geometry, such as the position of the geometry's vertices, is rotated about the vertical axis that passes through the center of the projection circle and corresponds to the viewer's position. The rotated geometry can then be rasterized, shaded conventionally, and written into the faces of the cube image. The degree of rotation can be determined by the distance to the object. The rotation should correspond to the projection circle obtained through raymarching. Fig. Figure 8 shows an example of how to handle translucency in stereo and adapt it to the projection circle.

[0049] Procedure 400 continues with step 450, in which post-translucency effects are applied to both cube images of the stereoscopic pair. Tone mapping is an example of the applicable post-translucency effects.

[0050] In step 460, projections for streaming are generated. These projections are created for both stereoscopic image pairs. Various types of projections can be generated for streaming, including equiangular, Boggs eumorphic, cube images, and pyramids. In step 470, the projections are then encoded and transmitted to client computing devices, such as head-mounted displays (HMDs). The process concludes in step 480.

[0051] Fig. Figure 5 illustrates a diagram of an example of an omnidirectional stereo projection in accordance with the principles of the invention. Fig. Figure 5 illustrates a projection circle 510 with a projection center 520 and several rays emanating from projection points on the projection circle 510, approximating a circular projection. To compute a regular perspective projection, the origin is the projection center 520 (eye position), and the direction can be calculated from the view and projection matrices and pixel coordinates. Using the disclosed rendering processes, a viewpoint can be shifted, tracing rays from several points on the projection circle 510. The original rendered image (non-circular) from the original perspective is used as source data for tracing rays from points on the projection circle 510. A couple of the projection points, 530 and 550, are shown in Figure 5. Fig. Figure 5 shows the projection point. Projection point 530 is a projection point for the left eye, and projection point 550 is a projection point for the right eye. The left and right projection points 530 and 550 correspond to the same line of sight and are opposite points on the projection circle 510.

[0052] One difference between calculating a regular perspective projection from projection center 520 and a stereoscopic panorama lies in the calculation of the desired ray origin and ray direction for each output pixel. To calculate the stereoscopic panorama, the direction can be calculated based on the output pixel coordinates within the panorama, and the ray origin is calculated based on the panorama center and the direction in which the ray origin lies on the projection circle 510. For example, projection point 530 or 550 can be the ray origin on the projection circle 510. Consequently, when calculating panoramic views, the disclosed method can determine a viewing direction for each output pixel, regardless of whether the projection or the layout of the rendered views is a cube or equilateral.For example, an equirectangular projection for x, y can be used to determine the viewing direction by defining pixel position geometry. In stereo panoramas, the projection point lies on the projection circle. For a given direction, the projection point is adjusted so that it lies on the projection circle 510.

[0053] Fig. Figure 6 illustrates a diagram 600, which provides an example of determining pixel colors for a left and a right eye with respect to a cube image in accordance with the principles of the invention. Diagram 600 illustrates the use of ODS to project rays in a cube image to determine the color for a left-eye pixel 633 and a right-eye pixel 635 at a single point on the surface of a monoscopic cube image 630. The colors of the left-eye pixel and the right-eye pixel for each point on each surface of the cube image 630 can also be determined. Advantageously, the ODS projection with raymarching can be used by a renderer to convert a rendered monoscopic cube image into a stereoscopic cube image. A renderer, such as the renderer 211 in Fig. 2 can be configured to perform the steps corresponding to the algorithm shown in diagram 600.

[0054] Diagram 600 includes a projection circle 610 with a projection center 620, which is positioned within a cube image 630. A single side of the surface 631 of the cube image 630 is in Fig. 6. At a single point on surface 631, there is a left-eye pixel 633 and a right-eye pixel 635. To determine which color should be visible to the left-eye pixel 633 and the right-eye pixel 635 at that point on surface 631, a projection line 640 is drawn from the point on surface 631 to the projection center 620. Projection points on the projection circle 610 for the left eye and the right eye are then determined by the intersection of a perpendicular line 642 on the projection circle 610. The perpendicular line 642 is perpendicular to the projection line 640 through the projection center 620. Projection points for the left eye 612 and the right eye 614 are located on the projection circle 610 where the perpendicular line 642 intersects the projection circle 610.Starting from the left-eye projection point 612 and the right-eye projection point 614, rays are traced in the same direction as that of the projection line 640 from the projection center 620 to the point on the front face 631 of the left-eye pixel 633 and the right-eye pixel 635. In other words, the rays from projection points 612 and 614 are parallel to the viewing direction of the projection line 641. When the rays strike a first object, the color of that object is used for the left-eye pixel 633 and the right-eye pixel 635, respectively. For example, if the left-eye ray 652 strikes object 662, the color of object 662 is used for left-eye pixel 633. Similarly, if the right-eye ray 654 strikes object 664, the color of object 664 is used for right-eye pixel 635. This process is repeated for every eye position, every viewing direction, and every pixel on the cube image.

[0055] Fig. Figure 7 illustrates a diagram 700 of an example for removing distortions from a stereoscopic cube image in accordance with the principles of the invention. Distortions can occur in some places when rays are traced from a projection circle with the IPD diameter. Different colors can be achieved for pixels that are close together in a cube image, leading to distortions from above and below, often referred to as spiral distortion. A renderer, such as renderer 211 in Fig. 2 can be configured to perform the steps corresponding to the algorithm shown in diagram 700.

[0056] To remove distortions from above and below, the diameter of the projection circle is gradually reduced from the original diameter of an IPD for horizontal rays to zero for vertical rays. To illustrate this gradual reduction, diagram 700 includes a first projection circle 710, a second projection circle 712, and a third projection circle 714. The horizontal rays 730 and 732 are traced from the first projection circle 710, which is the original projection circle with the original diameter of IPD. For rays that are traced upwards and downwards, collectively represented by rays 740 that go downwards in diagram 700, the diameter of the projection circle for these rays is gradually reduced.In diagram 700, this is represented by the rays 740, which are projected from the second projection circle 712 and the third projection circle 714, so that the rays appear as if projected from the same projection point.

[0057] Fig. Figure 8 illustrates a diagram 800 of an example of the application of translucency in stereo in accordance with the principles of the invention. Since unambiguous depth information is required to render translucency, it cannot be rendered into a cube image before raymarching. However, after raymarching, depth information is available for each surface, and the translucency can be rendered for a stereoscopic cube image as disclosed herein.

[0058] Diagram 800 includes a projection circle 810 with a projection center 820. Diagram 800 also includes an object 830, which is an original monoscopic object, and representations of the object's right and left eyes, designated as a right-eye object 840 and a left-eye object 850, respectively. When transforming the original object 830 into the right-eye object 840 and the left-eye object 850 using rays from the projection circle 810, the vertices of the original object 830 are rotated around the vertical axis to an extent dependent on the distance.

[0059] Diagram 800 illustrates an example of adjusting the position of vertices to convert a rendered monoscopic image into stereo. Before the conversion to stereo using the original object 830, the positions of the vertices of the original object 830 are adjusted by rotation so that each eye receives a different version of the object 830. For example, for the right eye, the vertices are shifted to the left, and for the left eye, the vertices are shifted to the right.

[0060] For example, a point on the original object 830, shown as p, is visible from a projection point 812, as shown by a first projection line 813 drawn from p to projection point 812. The radius from the projection center 820 to projection point 812 is ½ IPD. A second projection line 814, drawn from the projection center 820 and parallel to the first projection line 813, shows where point p will actually appear on the cube image. Therefore, point p must be rotated by an angle alpha around the central axis between point p and projection center 820 (p0). The sine of alpha is equal to the radius divided by the distance between point p and projection center 820, as shown by Equation 1 below. sin α=±IPD2‖p−p0‖

[0061] In contrast to conventional methods, the disclosed devices, schemes and rendering methods differ in that, in addition to color information, depth information is also used, which makes it possible to reproject the images to different viewpoints and not just in different viewing directions.

[0062] This is an improvement on traditional methods for capturing stereo panoramas, which typically involve stitching together a small number of images and discarding the resulting depth defects and stitching artifacts as simply unavoidable. The invention offers the advantage of rendering a monoscopic view first, and then converting that monoscopic view into a stereoscopic one. Regular rasterization can then be used to render projections without distortion.

[0063] Traditionally, several image processing algorithms are used that attempt to reduce stitching artifacts by analyzing images using computer vision algorithms, such as optical flow. While effective, these algorithms require many input images to achieve good results, and the analysis process is very time-consuming. Some VR HMD drivers can use a reprojection technique to adjust the images rendered by the application to a slightly newer orientation of the HMD. This can be described as a "time lag." However, only color information is used, and attempts are made to make the reprojection shader as fast as possible, so it can only correct the projection orientation and not the viewing angle.

[0064] The improved rendering process disclosed herein enables the real-time generation of a pair of stereoscopic images while rendering only a monoscopic set of images, such as a cube image. This keeps rendering costs relatively low and produces a panorama with a high-quality stereo effect. A two-layer cube image can be used for further quality improvement. In some embodiments, the disclosed rendering process can be used for Nvidia Ansel stereo panoramas by having the game provide an Ansel runtime with its depth buffer and depth projection parameters. Ansel is an advanced virtual photography tool for games that allows users to capture various types of images from within games, including regular screen captures, high-resolution screen captures, and mono and stereo 360-degree panoramas.With Ansel, a user can create screen recordings from any position, adjust them with post-processing filters, capture HDR images in high-fidelity formats, and share them in 360 degrees with their mobile phone, PC, or VR headset.

[0065] Accordingly, the system, devices, and methods described above, or at least a part thereof, may be embodied in or executed by various processors, such as digital data processors or computers, wherein the computers are programmed or store executable programs of sequences of software instructions to perform one or more of the steps of the methods. The software instructions of such programs disclosed herein may represent algorithms and be stored in machine-executable form on non-volatile digital data storage media, e.g.,The data may be encoded on magnetic or optical disks, random access memory (RAM), magnetic hard disks, flash memory and / or read-only memory (ROM) to enable different types of digital data processing systems or computers to perform one, several or all steps of one or more of the methods or components of the devices described herein.

[0066] Certain embodiments disclosed herein may further relate to computer storage products comprising a non-volatile, computer-readable medium on which program code for performing various computer-implemented operations is located, embodying the devices, systems, or steps of the methods described herein. The non-volatile medium used herein refers to all computer-readable media except volatile, propagating signals. Examples of non-volatile, computer-readable media include, but are not limited to: magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs; magneto-optical media such as floppy disks; and hardware devices specifically configured for storing and executing program code, such as ROM and RAM devices.Examples of program code include both machine code, as generated by a compiler, and files containing higher-level code that can be executed by the computer using an interpreter.

Claims

[1] Methods for providing stereoscopic images, comprising: Generating a monoscopic set of rendered images; and Converting the set of rendered images into a stereoscopic image pair using depth information from the monoscopic set of rendered images and raymarching, characterized by , that the monoscopic set of rendered images is a cube image (310) with both a color buffer and a depth buffer, and the stereoscopic image pair is a stereoscopic pair of cube images. [2] Method according to claim 1, wherein the conversion includes using an omnidirectional stereo (ODS) projection to project rays (340) onto a surface of the cube image to determine a color for left-eye pixels and a color for right-eye pixels for individual points on the surface of the cube image (310). [3] Method according to claim 2, wherein the raymarching includes determining surfaces on the surface intersecting with the rays (340) using the depth buffer. [4] Method according to claim 3, wherein the depth buffer includes mipmap planes and the raymarching includes using the mipmap planes to determine the surfaces on the surface intersecting with the rays. [5] Method according to claim 2, wherein the surface is one of six surfaces of the cube image (310) and the conversion includes the ODS projection, which projects rays onto each of the six surfaces to determine pixel colours of the left and right eye for points on each of the six surfaces using raymarching. [6] Method according to claim 5, wherein the raymarching selects the area to start with based on an origin of one of the rays. [7] Method according to claim 2, further comprising reducing a diameter of a projection circle of the ODS projection to remove spiral distortions of the stereoscopic pair of images. [8] Method according to claim 1, further comprising rendering translucent geometry for each of the stereoscopic pair of images after raymarching. [9] Method according to claim 8, further comprising reprojecting the pair of stereoscopic images for video transmission. [10] Method according to claim 1, wherein the monoscopic set of rendered images is a monoscopic set of rendered images of 360 degrees and the stereoscopic image pair is a stereoscopic image pair of 360 degrees. [11] Broadcaster for a virtual view (110), comprising: a cloud-based renderer (111) configured to generate a monoscopic set of rendered 360-degree images and to use raymarching and depth information to convert the set of rendered images into a stereoscopic 360-degree image pair, wherein the monoscopic 360-degree set of rendered images is a cube image (310) with both a color buffer and a depth buffer, and the stereoscopic 360-degree image pair is a stereoscopic cube image pair; and an image processor (112) configured to encode the stereoscopic image pair for video transmission. [12] Broadcaster for a virtual view (110) according to claim 11, further comprising a video transmitter (113) configured to transmit the stereoscopic image pair of 360 degrees as a video stream via a communication network (170). [13] Broadcaster for a virtual view (110) according to claim 12, wherein the cloud-based renderer (111) is configured to determine a color for left-eye pixels and a color for right-eye pixels for individual points on each of the faces of the cube image (310) by projecting rays using an omnidirectional stereo (ODS) projection onto each face of the cube image (310). [14] Broadcaster for a virtual view (110) according to claim 12, wherein the raymarching includes the use of the depth buffer to determine surfaces on each face of the cube image (310) that intersects with projected rays. [15] Broadcaster for a virtual view (110) according to claim 11, wherein the cloud-based renderer (111) is further configured to display a translucent geometry for each of the stereoscopic image pairs after raymarching. [16] Cloud-based renderer (111), comprehensive: a memory; and at least one processor coupled to the memory and configured to convert a cube image (310) into a stereoscopic pair of cube images using raymarching and a depth buffer of the cube image. [17] Cloud-based renderer (111) according to claim 16, wherein the processor is configured to determine a color for left-eye pixels and a color for right-eye pixels for individual points on each of the faces of the cube image (310) by projecting rays using an omnidirectional stereo (ODS) projection onto each face of the cube image (310). [18] Cloud-based renderer (111) according to claim 16, wherein the processor is further configured to render a translucent geometry for each of the stereoscopic cube image pairs after raymarching. [19] Cloud-based renderer (111) according to claim 16, wherein the at least one processor is further configured to generate the cube image (310).

Images