Offset Low-Discrepancy Spherical Sampling for Image Rendering

Abstract

An example image processing system includes a processor configured to receive a three-dimensional environment. The processor is configured to: apply a cube map function to a set of cube map texture coordinates to generate cube map directions; apply a low-discrepancy sequence sampling function to the cube map directions to generate low-discrepancy spherical sampling directions; and apply an offset cube mapping function to the low-discrepancy spherical sampling directions to generate offset sampling directions. The processor is configured to generate pixel values ​​for the cube map texture coordinates using the offset sampling directions and the three-dimensional environment.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 386,559 (Reference No. D22061USP1), filed December 8, 2022, and European Application No. 23166550.6 (Reference No. D22061EP), filed April 4, 2023, both of which are incorporated by reference in their entireties.

[0002] Various example embodiments relate to image processing operations, and more particularly, but not exclusively, to 360° image rendering. [Background technology]

[0003] Human visual acuity peaks at the fovea, where the concentration of cones is highest, and decreases with increasing retinal eccentricity. Foveated rendering refers to a set of computer graphics techniques in which the resolution of the rendered image is adjusted to match the visual acuity of the viewer's eye; that is, more detail is provided at the center of the viewer's gaze, with progressively less detail as one moves away from that point. This technique is implemented in virtual reality (VR) and augmented reality (AR) applications, as foveated rendering offers reductions in rendering time and data transmission bandwidth.

[0004] Cube map texturing is a technique for representing three-dimensional (3D) environment maps, such as 360° still images or video frames. In particular, a textured 3D cube is represented as six two-dimensional (2D) textures, one for each cube face. Modern graphics processing units (GPUs) provide hardware support for cube map texturing, including handling sampling across more than a single cube face. Summary of the Invention

[0005] Various embodiments of an image processing system for encoding and decoding spherically sampled data are disclosed herein. One example embodiment provides an image processing system including a processor for encoding data, the processor configured to receive a three-dimensional environment. For a set of cube-map texture coordinates, the processor is configured to: apply a cube-map function to the cube-map texture coordinates to generate cube-map directions; apply a low-discrepancy sequence sampling function to the cube-map directions to generate low-discrepancy spherical sampling directions; and apply an offset cube-mapping function to the low-discrepancy spherical sampling directions to generate offset sampling directions. The processor is configured to generate pixel values ​​for the cube-map texture coordinates using the offset sampling directions and the three-dimensional environment.

[0006] According to an example embodiment, an image processing system for encoding spherically sampled data is provided, the image processing system including a processor configured to receive a three-dimensional environment, apply an offset low discrepancy sequence (OLDS) sampling function to a set of cube-mapped texture coordinates to generate OLDS orientations, and generate pixel values ​​for the cube-mapped texture coordinates using the OLDS orientations and the three-dimensional environment.

[0007] According to another example embodiment, an image processing system for decoding spherically sampled data is provided. The image processing system includes a processor for decoding the data, the processor configured to receive an offset-sampled cube map from an encoded bitstream. The processor is configured to: apply an inverse offset cube mapping function to the offset sampling direction to generate a low-discrepancy spherical sampling direction; apply an inverse low-discrepancy sequence sampling function to the low-discrepancy spherical sampling direction to generate a cube map direction; and apply the inverse cube map function to the cube map direction to generate cube map texture coordinates. The processor is configured to generate pixel values ​​for the offset sampling direction using the cube map texture coordinates and the offset-sampled cube map. [Brief explanation of the drawings]

[0008] Other aspects, features, and advantages of the various disclosed embodiments will become more fully apparent, by way of example, from the following detailed description and the accompanying drawings.

[0009] [Figure 1] FIG. 1 illustrates an example process for a video distribution pipeline according to one embodiment. [Figure 2]FIG. 1 shows an example representation of the field of view of an average observer's eye. [Figure 3] FIG. 1 illustrates an example of a cube map in a 3D environment according to one embodiment. [Figure 4A] FIG. 1 illustrates an example of an offset cube map in a 3D environment according to one embodiment. [Figure 4B] FIG. 1 illustrates an example of an offset cube map in a 3D environment according to one embodiment. [Figure 4C] FIG. 1 illustrates an example of an offset cube map in a 3D environment according to one embodiment. [Figure 5A] FIG. 10 illustrates an example of uniformity of various sampling densities of a cube map. [Figure 5B] FIG. 10 illustrates an example of uniformity of various sampling densities of a cube map. [Figure 5C] FIG. 10 illustrates an example of uniformity of various sampling densities of a cube map. [Figure 6A] 10A-10C illustrate examples of uniformity of various sampling densities of offset cube maps. [Figure 6B] 10A-10C illustrate examples of uniformity of various sampling densities of offset cube maps. [Figure 6C] 10A-10C illustrate examples of uniformity of various sampling densities of offset cube maps. [Figure 7] FIG. 10 illustrates triangular regions from a Delaunay triangulation of an offset standard cube map spherical sampling. [Figure 8] FIG. 10 illustrates triangular regions from a Delaunay triangulation of an offset unicube cube-map spherical sampling. [Figure 9] 2 illustrates an example process performed by the video delivery pipeline of FIG. 1 according to one embodiment. [Figure 10] 2 illustrates another example process performed by the video delivery pipeline of FIG. 1 according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present disclosure and aspects thereof may be embodied in various forms, including hardware, devices, or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces, application programming interfaces, and hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. The foregoing is intended only to give a general idea of ​​various aspects of the present disclosure and is not intended to limit the scope of the present disclosure in any way.

[0011] Video applications as described herein may refer to any of video display applications, VR applications, AR applications, automotive entertainment applications, remote presence applications, display applications, gaming applications, mobile applications, internet-based video streaming applications, and the like.

[0012] The above technology is applicable to virtual reality cinema-immersive movie watching (e.g., for a head-mounted display, etc.) for streaming video data between a video streaming server and a video streaming client. Examples of video content may include, but are not necessarily limited to, any of audiovisual programs, movies, video programs, television broadcasts, computer games, AR content, VR content, automotive entertainment content, etc. Examples of video streaming clients may include, but are not necessarily limited to, any of display devices, computing devices with near-eye displays, head-mounted displays (HMDs), mobile devices, wearable display devices, set-top boxes with displays such as televisions, video monitors, etc.

[0013] As used herein, a "video streaming server" may refer to one or more upstream devices that prepare video content and stream it to one or more video streaming clients for rendering at least a portion of the video content (e.g., a portion corresponding to a user's FOV or viewport) on one or more (target) displays. The displays on which the video content is rendered may be part of or operate in conjunction with the one or more video streaming clients. Examples of video streaming servers may include, but are not necessarily limited to, any of a cloud-based video streaming server located remotely from the video streaming clients, a local video streaming server connected to the video streaming clients over a local wired or wireless network, a VR device, an AR device, an automotive entertainment device, a digital media device, a digital media receiver, a set-top box, a game console, a general-purpose personal computer, a tablet, a dedicated digital media receiver, etc.

[0014] Video encoding according to example embodiments 1 illustrates an example process for a video delivery pipeline 100, showing various stages from video capture to video content display, according to one embodiment. A sequence of video frames 102 may be captured or generated using an image generation block 105. The video frames 102 may be digitally captured (e.g., by a digital camera) or computer-generated (e.g., using computer animation) to provide video data 107. Alternatively, the video frames 102 may be captured on film by a film camera. The film may then be translated into a digital format to provide video data 107.

[0015] In the production phase 110, the video data 107 may be edited to provide a video production stream 112. The data in the video production stream 112 may then be provided to a processor (or one or more processors, such as a central processing unit (CPU)) in a post-production block 115 for post-production editing. The post-production editing in block 115 may include, for example, adjusting or modifying the color or brightness of specific areas of the image to enhance image quality or achieve a particular look of the image according to the videographer's creative intent. This portion of post-production editing is sometimes referred to as "color timing" or "color grading." Other edits (e.g., scene selection and sequencing, image cropping, adding computer-generated visual special effects, etc.) may be performed in block 115 to produce a "final" version 117 of the work for distribution. For example, rendering techniques described herein, such as cube mapping, cube map offset, and sampling techniques, may be performed during post-production editing 115. During post-production editing 115, the video image may be viewed on a reference display 125. In other examples, the rendering techniques described herein may occur in the image generation block 105, the production phase 110, or any other suitable processing step within the video delivery pipeline 100.

[0016] Following post-production 115, the final version 117 of video data may be delivered to an encoding block 120 for downstream distribution to decoding and playback devices such as VR headsets, near-eye displays, etc. In some embodiments, encoding block 120 may include audio and video encoders, such as those defined by ATSC, DVB, DVD, Blu-Ray, and other distribution formats, to generate a coded bitstream 122. At the receiver, coded bitstream 122 is decoded by a decoder 130 to generate a corresponding decoded signal 132, which represents a copy or approximation of signal 117. The receiver may be attached to a target display 140, which may have slightly or completely different characteristics than the reference display 125. In such cases, a display management (DM) block 135 may be used to map the decoded signal 132 to the characteristics of the target display 140 by generating a display-mapped signal 137. Depending on the embodiment, the decoder 130 and the display management block 135 may comprise separate processors or may be based on a single integrated processing unit. The encoded bitstream 122 may include metadata to aid in the reconstruction of the final version 117.

[0017] Human field of vision Techniques such as those described herein may be implemented based on knowledge of the (human) eye and how it perceives brightness, color, spatial resolution, etc. These techniques may be applied based on the particular visual characteristics of an individual observer or the average or collective visual characteristics of a user population.

[0018] Figure 2 shows an example representation of the visual field of an average observer's eye. As shown in Figure 2, the distribution of cones and rods (in the eye) may be divided into different distribution ranges of cones and rods and further projected into an angular visual field representation (of the eye). The highest level of visual perception is achieved in the foveal (field of view) region 202 of the eye.

[0019] By way of example and not limitation, the widest angular range of an eye's field of view, without considering visual constraints due to facial anatomy, may be along the horizontal direction in FIG. 2, parallel to the interpupillary line between the observer's eyes, which may be approximately 180 angular degrees.

[0020] Each of the concentric circles (e.g., labeled 30°, 60°, 90°, etc.) represents a direction at an equal (or the same) angle relative to the view direction of the observer's left or right eye. Note that the angles of 30°, 60°, 90°, etc. are merely illustrative. Different values ​​of the angles or different sets of angles can be used to define or describe the observer's field of view. The view direction (not shown in FIG. 2 ) is oriented vertically from the plane of FIG. 2 at the intersection of the transverse direction 212 and the vertical direction 214 in the foveal region 202 (darkest fill pattern). Here, the transverse direction 212 and the vertical direction 214 form a plane normal to the view direction.

[0021] As shown in FIG. 2, the field of view of the eye may be divided into a foveal region 202 (e.g., logically, projected by a particular division in the rod / cone density distribution, etc.) immediately surrounded by a parafoveal region 204. In some embodiments, the foveal region 202 may correspond to the observer's foveal vision and may extend from zero (0) degrees to a first angle (e.g., 2-4 degrees, 3-7 degrees, 5-9 degrees, etc.) relative to the viewing direction. In some embodiments, the parafoveal region 204 may extend from a first angle to a second angle (e.g., 6-12 degrees, etc.) relative to the viewing direction.

[0022] The paracentral region 204 is immediately surrounded by the near peripheral region 206. The near peripheral region 206 is immediately adjacent to the mid peripheral region 208, which is immediately adjacent to the far peripheral region 210, which is the remainder of the field of view. In some embodiments, the near peripheral region 206 may extend from a second corner to a third corner (e.g., 25-35 degrees, etc.) relative to the viewing direction. In some embodiments, the mid peripheral region 208 may extend from the third corner to a fourth corner (e.g., 50-65 degrees, etc.) relative to the viewing direction. The far peripheral region 210 may extend from the fourth corner to the edge of the field of view.

[0023] The first, second, third, and fourth angles used in this example logical division of the field of view may be defined or specified along a lateral direction 212. When the field of view of Figure 2 corresponds to a forward level viewing direction, the lateral direction 212 may be the same as or parallel to the interpupillary line of the observer.

[0024] In addition to or instead of the method shown in FIG. 2, in which the observer's field of view is logically divided into regions such as the foveal region, parafoveal region, near peripheral region, mid-peripheral region, and far peripheral region based on the angles, a different method of logically dividing the observer's field of view may be used.

[0025] For example, in some embodiments, the observer's field of view may be divided into more or fewer regions, such as a combination of foveal, near-peripheral, and far-peripheral regions, without para-foveal and / or mid-peripheral regions, etc. Such a logical division of the observer's field of view may use spatially faithful (or high fidelity) image portions that cover the foveal region through some or all of the near-peripheral region.

[0026] In some embodiments, the observer's field of view may be divided based on quantities other than angle, as shown in FIG. 2 . For example, in a non-limiting implementation, the foveal region may be defined as the field of view corresponding to the observer's foveal field of vision. The parafoveal region may be defined as the field of view corresponding to regions of the observer's retina where the cone / rod densities exceed a relatively high cone / rod density threshold. The near-peripheral region may be defined as the field of view corresponding to regions of the observer's retina where the cone / rod densities do not exceed a relatively high cone / rod density threshold, but exceed an intermediate cone / rod density threshold. The mid-peripheral region may be defined as the field of view corresponding to regions of the observer's retina where the cone / rod densities do not exceed an intermediate cone / rod density threshold, but exceed a relatively low cone / rod density threshold. A focal-vision region as described herein may cover part or all of the region from the observer's foveal vision (e.g., part or all of the observer's near-peripheral vision, etc.) based on thresholds that are not necessarily angular-based (e.g., cone density / rod density thresholds, etc.).

[0027] Additionally, optionally, or alternatively, a combination of two or more different ways of logically dividing the observer's field of view and / or other human vision factors may be used to determine the focal field region of the observer's field of view. For example, instead of using a focal field region as described herein to cover the same range of angular values ​​in different angular directions, a focal field region as described herein may cover a larger range of angular values ​​along the horizontal direction 212 than the range of angular values ​​covered by the focal field region along the vertical direction 214. This is because the human visual system may be more sensitive to image details along the horizontal direction 212 than to image details along the vertical direction 214.

[0028] In some embodiments, the focal vision region as described herein covers some or all of the foveal region (e.g., plus a safety margin, etc.), the paracentral region (e.g., extending from the foveal region, excluding the foveal region, etc.), the near-peripheral region (e.g., extending from the paracentral region, further excluding the paracentral region, etc.), the mid-peripheral region (e.g., extending from the near-peripheral region, further excluding the near-peripheral region, etc.), etc.

[0029] Additionally, optionally, or alternatively, in some embodiments, the field of view of an eye as described herein takes into account vision-related factors such as eye swiveling, viewing constraints from the nose, cornea, eyelids, etc.

[0030] Examples of focal field regions as described herein include, but are not necessarily limited to, any combination of one or more of a circular, oblong, elliptical, heart, star, round, square, polygonal shape, and the like.

[0031] Gaze tracking In some embodiments, only a (e.g., relatively small) focal field region of the eye's field of view needs to be provided with pixel values ​​with the highest dynamic range, widest color gamut, highest (or sharpest) spatial resolution, etc. In some embodiments, the focal field region of the eye's field of view may correspond approximately to some or all of the eye's near-peripheral field of view, as opposed to the entire foveal field of view of the eye. In some embodiments, the focal field region of the eye's field of view may additionally include a safety vision field region.

[0032] In some embodiments, the size and / or shape of the safety viewing region in the focal viewing region may be preset to a fixed size (e.g., 0%, 5%, 10%, −5%, −10%, etc.) that does not change depending on network bandwidth, image content, the type of computing device (e.g., helmet-mounted display device, wall display, etc.) participating in the video application, the type of rendering environment (e.g., cloud-based video streaming server, etc.) participating in the video application, etc. In some other embodiments, the size and / or shape of the safety viewing region in the focal viewing region may be dynamically reconfigurable at runtime and its range may vary. For example, in response to determining that the network connection does not support a relatively high bandwidth, the size and / or shape of the safety viewing region may be dynamically reduced at runtime from 10% over the foveal field of view of the eye to 5%. On the other hand, in response to determining that the network connection does support a relatively high bandwidth, the size and / or shape of the safety viewing region may be dynamically expanded at runtime from 5% over the foveal field of view of the eye to 10% over the foveal field of view of the eye.

[0033] The size and / or shape of the safe viewing area may be set depending on the latency in eye tracking. Additionally, optionally, or alternatively, eye tracking data as described herein may be used to predict where the observer will look next and reduce bandwidth / safety areas based on the prediction. For example, a user's viewing direction at runtime may be tracked by a viewing direction tracking device. The viewing direction tracking device may operate in real time with a display on which a sequence of display mapped images is rendered. As the user changes their viewing direction and / or viewing distance from moment to moment, the viewing direction tracking device, as described herein, tracks and calculates the viewing angle and / or viewing distance in the coordinate system in which the sequence of display mapped images is rendered, generates a time-based sequence of viewing directions, and transmits each viewing direction in the time-based sequence of viewing directions as a signal to the video streaming server. Each of such signaled viewing directions of the observer received by the video streaming server may be indexed by a time point value. The time instant values ​​may be associated or correlated with the offset LDSS cubemap by a video streaming server as described herein.

[0034] While an observer is viewing the 3D environment or a derivative version, view direction data is collected to track the observer's view direction. Examples of view direction data may include, but are not limited to, linear displacement, angular displacement, linear or translational motion, angular motion or rotation, pitch, roll, yaw, sway, heave, surge, etc., which may be collected by any combination of eye-tracking devices, face-tracking devices, FOV-tracking devices, etc. The observer's view direction at multiple points in time may be determined. The observer's view direction can be used on the recipient device (or a device operating in conjunction with the recipient device) to generate a new view by implementing some or all of the techniques described herein on the recipient device itself until an upstream device responds with new data.

[0035] View direction data may be collected, analyzed, and / or shared / transmitted between view direction tracking devices and streaming devices with relatively low latency (e.g., within a fraction of an image frame time, within 5 milliseconds, etc.). In an example implementation, view direction tracking data may be shared between these devices using the lowest latency data / network connection of multiple data / network connections available. View direction data is sometimes referred to herein as a gaze vector.

[0036] In response to determining the view direction of the observer relatively quickly and with relatively low latency (e.g., within one fraction of an image frame time) based on the view direction data, the video streaming server may dynamically reduce the size and / or shape of the safe viewing region at runtime from 10% to 5% over the foveal field of view of the eye. A relatively small region (e.g., within 20 degrees of the view direction), such as the highest dynamic range, widest color gamut, highest spatial resolution, etc., may be transmitted in the video signal to downstream receiving devices.

[0037] On the other hand, in response to determining the view direction of the observer relatively late (e.g., above a time threshold, above a fraction of one image frame time, above 5 milliseconds, etc.) based on the view direction data, the video streaming server may dynamically expand the size and / or shape of the safe viewing area at runtime by 1% to 3%, 2% to 6%, 5% to 10%, etc. across the foveal field of view of the eye. A relatively wide area (e.g., up to a 30-degree angle from the view direction), such as the highest dynamic range, widest color gamut, or highest spatial resolution, may be transmitted in the video signal to a downstream receiving device. In this way, the receiving device of the video signal may have sufficient image data across a relatively large focal field of view area to make localized decisions based on the view direction for image rendering.

[0038] Cube Mapping Technology The embodiments described herein relate to a 3D virtual environment generated by a computer graphics engine or captured using one or more cameras. The 3D virtual environment is then accessed using an immersive device such as a virtual reality headset. Foveated rendering techniques may be implemented to assist in rendering images with maximum resolution at the center of the observer's line of sight. One technique for representing a 3D environment in one or more 2D textures includes mapping the 3D virtual environment to a cube, thereby generating a cube map. As an example, in a computer graphics program operating on the instantaneous state of a 3D model map (e.g., a scene), six virtual cameras are constructed with a common focal point. The six cameras look along the positive and negative directions of three orthogonal axes, each pointing toward the center of a face of the cube. The common focal point may be located anywhere within the scene, and the arrangement of the six cameras may be arbitrarily oriented within the scene.

[0039] FIG. 3 provides an example of a cube mapping technique for a 3D environment (e.g., a scene). In the example of FIG. 3, the 3D environment 300 includes a texture cube map 305. The center of the cube map 305 is located at an arbitrary point within the 3D environment 300. The 3D environment 300 is mapped onto the cube map 305. For example, the cube map 305 includes a first face 315. A camera view 310 (e.g., a virtual camera) is located at the center of the cube map 305 and faces perpendicular to and outward toward the center of the first face 315. The portion of the 3D environment 300 captured by the camera view 310 is mapped onto the first face 315. This process is repeated for each face of the cube map 305 until the entire 3D environment 300 is mapped onto the cube map 305.

[0040] For the example six-camera arrangement, a virtual cube (e.g., cube map 305) encloses the six-camera arrangement, with the center of cube map 305 coinciding with the focal points of the cameras. The focal points of the six cameras may be anywhere within the scene and are not necessarily located at the center of the scene. Three orthogonal axes pass through the center of a face (e.g., first face 315) of cube map 305. Uniform sampling of each face of the virtual cube generates six images that form texture cube map 305. Each image is an image of the scene captured by one of the six cameras. In some examples, the field of view of each camera is 90 degrees.

[0041] However, cube mapping can result in spherical samples (or 3D model maps) being more densely concentrated near the spherical projections of the cube map's edges and corners, and less densely concentrated at the projections of the cube's face centers. This results in higher resolution (in pixels per solid angle) at the corners and edges of the cube, and lower resolution at the center of each face (relative to the edges and corners). This non-uniformity causes area and angle distortions and artifacts in the environment map when the cube map is rendered into a viewport. Furthermore, when rendering video, standard cube map sampling wastes rendering time and bandwidth in directions away from the viewer's line of sight.

[0042] To address this issue, offset cube mapping techniques may be implemented. Offset cube mapping is a foveated rendering technique in which a virtual camera used to generate the six faces of a cube-map texture is biased or offset along one axis. This offset increases detail in the direction of the offset and decreases detail in the opposite direction. For example, FIGS. 4A-4C provide an example of offset cube mapping for a 3D environment. FIG. 4A shows a 3D environment 400 including a first virtual camera 405 and a six-sided texture cube map 412, four of which (front face 410, back face 415, and side face 420) are shown in FIG. 4A. FIG. 4B shows a second virtual camera 425 in the 3D environment 400 offset by a bias distance 427 (e.g., an offset bias) and a six-sided texture offset cube map 432, four of which (front face 430, back face 435, and side face 440) are shown in FIG. 4B. Bias distance 427 may be a vector having a magnitude and direction that points away from the common center of texture cube map 412 and texture offset cube map 432. The common center of texture cube map 412 and texture offset cube map 432 is located at a general point within 3D environment 400. Offsetting second virtual camera 425 places second virtual camera 425 closer to the edge of 3D environment 400 than first virtual camera 405. The first camera view maps portions of the 3D environment onto the front face 410 of texture cube map 412, and the offset camera view maps portions of 3D environment 400 onto the front face 430 of texture offset cube map 432.

[0043] 4C shows that second virtual camera 425 captures a narrower view 445 of environment 400 on front side 430 relative to first virtual camera 405, as indicated by the smaller range of angles subtended by 430 relative to 410, thereby providing a higher level of detail in the frontal direction. Similarly, an offset back-facing camera (not shown) captures a wider view 450 of environment 400 relative to back side 435 than a back-facing standard cube map camera (not shown), thereby providing a lower level of detail in that direction.

[0044] For the example six-camera arrangement, a virtual cube (e.g., first texture cube map 405) surrounds the six-camera arrangement, as described above. However, the center of the cube is offset from the common camera focal point along one of the orthogonal axes. Thus, each camera's view of the scene is different, with some cameras having narrower views of the scene and others having wider views. When each camera samples its corresponding cube face with the same sampling density, the cameras with narrower views of environment 400 sample the scene more densely than the cameras with wider views.

[0045] Functionally, offset cube mapping is performed by applying a cube map texture coordinate transformation that is applied in the shader before performing the cube map sampling function. By reducing the pixels rendered in directions away from the observer's gaze vector offset cube map, both rendering time and required bandwidth are reduced. However, offset cube mapping can experience warping of spatial samples due to the offset.

[0046] Low discrepancy spherical sampling The embodiments described herein provide for applying offset cube-map virtual camera bias to low discrepancy spherical sampling (LDSS). Discrepancy, as described herein, refers to a measure of the uniform distribution of points on a sphere, with lower discrepancy values ​​suggesting a more uniform sampling. Further details regarding the uniform distribution of points on a sphere are provided in J. Cui, et al., “Equidistribution on the Sphere,” SIAM Journal on Scientific Computing, 1997, Vol. 18, No. 2: pp. 595-609, incorporated herein by reference. The LDSS technique described herein exhibits lower discrepancy, area, and angle distortion than standard (e.g., non-offset) cube mapping (e.g., the cube mapping shown in FIG. 5).

[0047] One example of an LDSS technique is isocube sampling. Isocube sampling involves first dividing a sphere into four equatorial zones and two polar zones, each of equal area. Each zone is then divided into NxN elements by multiple longitudinal and latitudinal curves. The resulting NxN elements, when mapped onto a cube, are relatively curved (e.g., warped) compared to samples in a standard cubemap. Samples on an isocube map are indexed on rings running parallel to the equator of the sampled sphere. Further details regarding isocube sampling are described in "L. Wan, et al., Isocube Exploiting the Cube map Hardware, IEEE Transactions on Visualization and Computer Graphics, July / August 2007, Vol. 13, No. 4: pp. 720-731," which is incorporated herein by reference.

[0048] Another example of an LDSS technique is unicube sampling. Unicube sampling involves dividing a cube face based on two sets of parallel grid lines. The parallel lines are distributed non-uniformly so that the division on the projected spherical surface maintains a uniform rectilinear structure. The non-uniform division of the cube face is then mapped to a uniform division. Further details regarding unicube sampling are described in "Tze-Yiu Ho, et al., Unicube for Dynamic Environment Mapping, IEEE Transactions on Visualization and Computer Graphics, January 2011, Vol. 17, No. 1: pp. 51-63," which is incorporated herein by reference.

[0049] LDSS technology provides improved uniformity of sampling density compared to the standard cube mapping of FIG. 3. FIG. 5A shows a unicube cube map 500 with a unicube sampling density. FIG. 5B shows a standard cube map 510 with a non-uniform sampling density (e.g., without LDSS technology applied). FIG. 5C shows an isocube cube map 520 with an isocube sampling density. The sampling of the unicube cube map 500 and the isocube cube map 520 is more uniform compared to the standard cube map 510.

[0050] Just as LDSS techniques provide improved sample distributions relative to standard cube map sampling, LDSS techniques also improve the sample distribution of offset cube maps (offset LDSS cube maps). For example, FIG. 6A shows a spherical projection of an offset unicube cube map 600 having both unicube sampling density and an offset bias. FIG. 6B shows a spherical projection of an offset standard cube map 610 having both standard cube map sampling density and an offset bias. FIG. 6C shows a spherical projection of an offset isocube cube map 620 having both isocube sampling density and an offset bias. The sampling of offset unicube cube map 600 and offset isocube cube map 620 is more uniform compared to offset cube map 610 (ignoring the intentional spatial warping of sample coordinates due to the offset). Therefore, implementing both offset bias and LDSS techniques improves the resulting spatial sample distribution compared to using offset bias alone. Additionally, implementing both offset bias and LDSS techniques results in better image quality, reduced transmission bandwidth requirements, and (in some examples) better video compression rates.

[0051] As a further example of the benefits of offset LDSS cube maps, Figure 7 shows the area of ​​triangles from a Delaunay triangulation of an offset cube map without LDSS technology applied. As shown in Figure 7, triangles with smaller areas (and therefore higher resolution) are located along the spherical projection areas corresponding to the edges and corners of the cube map faces. For comparison, Figure 8 shows the area of ​​triangles from a Delaunay triangulation of an offset unicube cube map. Compared to Figure 7, the area of ​​triangles in Figure 8 decreases monotonically as one approaches the center of the offset face.

[0052] Referring to the six-camera example arrangement and Figure 4, before the cubemap was offset, the cameras had a less uniform sampling density. After implementing LDSS technology, the cameras have a more uniform sampling density. The sample positions are changed to provide a more uniform distribution of samples when the sample directions are projected onto the surface of a concentric sphere.

[0053] Both offset cube mapping and LDSS techniques may be viewed as functions that take an input sample direction (expressed either in 3D Cartesian coordinates or 2D spherical angles) and produce a modified direction as output. For example, let x = (x, y, z) be the unit length direction vector. A direction vector as used herein refers to a 3-element vector in a 3D environment. A direction vector may also be simply referred to as a direction (e.g., unit length direction, sampling direction, etc.). An equivalent or corresponding spherical angle may be calculated from x. Let u = (u, v, f) be the cube map texture coordinates (where (u, v) are the horizontal and vertical texture coordinates and f is the face index in the range [1, 6]). u = cm -1Let cm(u)=x, with the inverse correspondence denoted as (x), define the correspondence between a direction vector and a standard cube map texture coordinate.

[0054] Let d be an axis-aligned offset direction of the form (d, 0, 0), (0, d, 0), or (0, 0, d), where d is a scalar value in the range (-1, 1). The offset cube mapping ocm(x, d) takes a direction vector x and an offset d and maps it to the modified direction vector x o When applied to cubemap texture coordinates, the modified offset cubemapping sample direction is x o (u,d)=ocm(cm(u),d), and the offset cube map pixel at u (i.e., the pixel at position (u,v) on face f) is the pixel in the direction x o This is an environmental sample.

[0055] Similarly, low discrepancy sequence sampling is x L = lds(x), where lds() represents unicube, isocube, or other LDSS methods. The offset low discrepancy sequence (OLDS) sampling function x c Both the LDSS and offset direction modifiers may be applied via function composition to obtain the offset low discrepancy sequence (OLDS) sampling function x = olds(u,d) = ocm(lds(cm(u),d)). c=olds(u,d)=ocm(lds(cm(u),d)) states that, given an offset d, the pixel at u in the OLDS texture is a scene sample in the direction ocm(lds(cm(u),d)). The OLDS texture map is obtained by sampling the scene over a discrete set of values ​​for u. This OLDS texture may be encoded and transmitted to the receiver. The offset d is also transmitted to the receiver, or an agreed-upon value is used by both the sender and receiver.

[0056] When a receiver is rendering a scene, for example for display in an AR or VR headset, the receiver obtains samples of the scene in multiple directions. To obtain a scene sample in direction x* from a received OLDS texture, the OLDS texture coordinate u* is determined by solving equation (1) for u*. x* = olds(u*,d) = ocm(lds(cm(u*),d)) (formula)(1)

[0057] To solve equation (1) for u*, the inverse functions lds(), ocm(), and cm() are applied. The inverse functions are denoted by the superscript -1. First, apply ocm() to both sides of (1). -1 () is applied to undo the offset mapping and generate the LDSS direction. ocm -1 (x*) = lds(cm(u*),d)

[0058] Next, lds -1 Apply () to both sides to get the standard cubemap orientation. lds -1 (ocm -1 (x*),d) = cm(u*)

[0059] Finally, cm -1 Apply () to both sides to undo the cubemap orientation and get the cubemap texture coordinate u*. cm -1 (lds-1 (ocm -1 (x * ),d)) = u *

[0060] 9 shows a method 900 for generating pixel values ​​for an offset sampling direction. Method 900 may be performed by an electronic processor included in post-production block 115, an electronic processor included in image generation block 105, an electronic processor included in production block 110, or the like. In step 902, the electronic processor receives a 3D environment. The 3D environment may be, for example, a 3D environment map, a 3D game engine scene, a ray-traced environment (e.g., ray tracer), a texture-mapped sphere, and other virtual 3D environments that can be mapped to a cube map.

[0061] In step 904, for a set of cube map texture coordinates, the electronic processor applies a cube map function to the cube map texture coordinates to generate a cube map direction. For example, for cube map texture coordinates u=(u,v,f), the electronic processor applies the cube map function cm() to obtain a cube map direction x=cm(u).

[0062] In step 906, the electronic processor applies a low-discrepancy sequence sampling function to the cube map direction to generate a low-discrepancy spherical sampling direction. For example, the electronic processor may apply a low-discrepancy sequence sampling function lds() to the cube map direction x to generate a low-discrepancy spherical sampling direction x. L In step 908, the electronic processor applies the offset cube mapping function to the low discrepancy spherical sampling direction to generate the offset sampling direction. For example, the electronic processor applies the offset cube mapping function ocm() to the low discrepancy spherical sampling direction x L(given offset d) to offset sampling direction x o =ocm(x L , d) is obtained.

[0063] In step 910, the electronic processor generates pixel values ​​for cube-mapped texture coordinates using the offset sampling direction and the 3D environment. For example, applying the cube-map function cm(), the low-discrepancy sequence sampling function lds(), and the offset cube mapping function ocm() yields x C =olds(u,d)=ocm(lds(cm(u),d)), indicating that for a given offset d, the pixel at u in the OLDS texture is a scene sample in the direction ocm(lds(cm(u),d)). In some examples, the pixel values ​​and / or the OLDS texture are encoded and transmitted to the receiver.

[0064] 10 shows a method 1000 for decoding an OLDS texture (e.g., an offset-sampled cube map). Method 1000 may be performed by an electronic processor included in decoding block 130. In step 1002, the electronic processor receives the offset-sampled cube map. In some examples, decoding block 130 decodes encoded bitstream 122 to obtain the offset-sampled cube map.

[0065] In step 1004, the electronic processor applies an inverse offset cube mapping function to the offset sampled cube map for the offset sampling direction to generate a low discrepancy spherical sampling direction. For example, the encoded bitstream 122 may include metadata to assist in the reconstruction of a 3D environment. In some examples, the metadata may include ocm -1 The electronic processor includes the distance and direction of the bias distance 427, which defines the ocm -1 () offset sampling direction x C, which reverses the distance and direction of bias distance 427, restoring bias distance 427 to x L get.

[0066] In step 1006, the electronic processor applies an inverse low-discrepancy sequence sampling function to the low-discrepancy sphere sampling directions to generate cube-map directions. For example, the electronic processor may apply the lds function to the low-discrepancy sphere sampling directions, as described above. -1 () is the inverse of the low discrepancy sequence sampling direction x L to obtain a standard cube map direction x. In step 1008, the electronic processor applies an inverse cube mapping function to the cube map direction to generate cube map texture coordinates. For example, the electronic processor may apply cm -1 () is applied to the cube map direction x to obtain the cube map texture coordinate u. In step 1010, the electronic processor uses the cube map texture coordinate and the offset sampled cube map to generate pixel values ​​in the offset sampling direction.

[0067] For interactive content such as VR and AR, the bias vector (d) (e.g., magnitude and / or direction) may be dynamically aligned with the gaze vector of the observer's viewing direction to maximize rendered image quality. Additionally, the magnitude of the bias vector may be varied based on the rate of change of the gaze vector. For example, when the viewing direction is moving rapidly, the magnitude of the bias vector may be reduced so that visual quality is more consistent across rapidly changing viewing angles. In one example of a fast-moving viewing direction, the virtual camera used to render left-eye and right-eye images in a head-mounted display quickly changes its orientation in response to rapidly changing user events such as head rotation, controller manipulation, and changes in gaze direction. In some implementations, the magnitude of the bias vector (e.g., the amount of offset) is determined based on a linear function. In other implementations, the magnitude of the bias vector is increased as the rate of change of the gaze vector increases above a threshold. For example, when the rate of change is below a first threshold, a first bias vector magnitude is selected. When the rate of change is above the first threshold but below a second threshold, a second bias vector magnitude is selected. When the rate of change exceeds a second threshold, a third bias vector magnitude is selected.

[0068] Additionally, for linear cinematic 360° content, the magnitude of the bias vector may be varied based on the likelihood that the viewer is facing a particular direction or based on the importance of the action in a particular direction. For example, for video content that includes a storyline, it is likely that the viewer is looking in a particular direction of the story. The axis of the offset (e.g., the direction of the bias vector) may be aligned with those directions, and increasing the magnitude of the bias vector in those directions improves the visual quality for important areas.

[0069] Although the embodiments described herein primarily refer to full 360° environments, in some instances, environments less than a full 360° may be rendered. For example, a subset of six cube faces (e.g., five faces) may be rendered. As another example, a sub-region within a face may be rendered.

[0070] The above video distribution systems and methods may provide image processing and 360° image rendering. Systems, methods, and devices according to the present disclosure may take any one or more of the following configurations:

[0071] (1) An image processing system for encoding spherically sampled data, the image processing system including a processor for encoding the data, the processor configured to: receive a three-dimensional environment; for a set of cube-mapped texture coordinates, apply a cube-map function to the cube-mapped texture coordinates to generate cube-mapped directions; apply a low-discrepancy sequence sampling function to the cube-mapped directions to generate low-discrepancy spherical sampling directions; apply an offset cube-mapping function to the low-discrepancy spherical sampling directions to generate offset sampling directions; and generate pixel values ​​for the cube-mapped texture coordinates using the offset sampling directions and the three-dimensional environment.

[0072] (2) The image processing system according to (1), wherein the low discrepancy spherical sampling operation is a unicube sampling operation.

[0073] (3) The image processing system according to (1), wherein the low discrepancy spherical sampling operation is an isocube sampling operation.

[0074] (4) An image processing system according to any one of (1) to (3), wherein the processor is further configured to provide pixel values ​​of cube-map texture coordinates via a virtual reality device.

[0075] (5) An image processing system as described in any one of (1) to (4), wherein the processor is configured to apply the offset cube mapping function by performing operations including receiving a gaze vector indicating a user's line of sight, determining a rate of change of the gaze vector, and selecting an offset value based on the rate of change of the gaze vector.

[0076] (6) An image processing system as described in (5), wherein selecting an offset value includes selecting a first offset value when the rate of change of the gaze vector is less than a first threshold, and selecting a second offset value when the rate of change of the gaze vector is greater than or equal to the first threshold, wherein the first offset value is greater than the second offset value.

[0077] (7) An image processing system described in any one of (1) to (6), wherein the processor is further configured to receive a gaze vector indicating a user's line of sight direction and align the offset sampling direction with the gaze vector.

[0078] (8) The image processing system according to any one of (1) to (7), wherein the processor is further configured to select a sampling frequency of the low discrepancy sequence sampling function.

[0079] (9) The image processing system according to any one of (1) to (8), wherein the processor is configured to apply a cube map function to cube map texture coordinates in a shader.

[0080] (10) An image processing system for encoding spherically sampled data, the image processing system including a processor for encoding the data, the processor configured to receive a three-dimensional environment, apply an offset low discrepancy sequence (OLDS) sampling function to a set of cube-mapped texture coordinates to generate OLDS orientations, and generate pixel values ​​for the cube-mapped texture coordinates using the OLDS orientations and the three-dimensional environment.

[0081] (11) The image processing system according to (10), wherein the OLDS sampling function includes a unicube sampling function.

[0082] (12) The image processing system according to (10), wherein the OLDS sampling function includes an isocube uniform sampling function.

[0083] (13) An image processing system according to any one of (10) to (12), wherein the processor applies the OLDS sampling function by performing operations including receiving a gaze vector indicating a user's direction of view, determining a rate of change of the gaze vector, selecting an offset value based on the rate of change of the gaze vector, and applying an offset cube mapping function using the offset value.

[0084] (14) The image processing system described in (13), wherein selecting an offset value includes selecting a first offset value when the rate of change of the gaze vector is less than a first threshold, and selecting a second offset value when the rate of change of the gaze vector is greater than or equal to the first threshold, and the first offset value is greater than the second offset value.

[0085] (15) An image processing system according to any one of (10) to (14), wherein the processor is further configured to receive a gaze vector indicating a user's line of sight direction, and align an offset sampling direction of the OLDS sampling function with the gaze vector.

[0086] (16) The image processing system according to any one of (10) to (15), wherein the processor is further configured to select a sampling frequency of the OLDS sampling function.

[0087] (17) An image processing system for decoding spherically sampled data, the image processing system including a processor for decoding the data, the processor being configured to: receive an offset sampled cube map from an encoded bitstream; apply an inverse offset cube mapping function to the offset sampling direction to generate a low-discrepancy spherical sampling direction; apply an inverse low-discrepancy sequence sampling function to the low-discrepancy spherical sampling direction to generate a cube map direction; apply the inverse cube map function to the cube map direction to generate cube map texture coordinates; and generate pixel values ​​for the offset sampling direction using the cube map texture coordinates and the offset sampled cube map.

[0088] (18) The image processing system according to (17), wherein the inverse low discrepancy sequence sampling function is an inverse unicube sampling function.

[0089] (19) The image processing system according to (17), wherein the inverse low discrepancy sequence sampling function is an inverse isocube sampling function.

[0090] (20) The image processing system of any one of (17) to (19), wherein the processor is further configured to receive metadata associated with the offset-sampled cube map from the encoded bitstream, the metadata including an inverse offset cube mapping function.

[0091] With respect to processes, systems, methods, heuristics, etc. described herein, although steps of such processes, etc., are described as occurring according to a particular order, it should be understood that such processes may be practiced by performing the steps described herein in an order other than the order described herein. Furthermore, it should be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the process descriptions herein are provided for the purpose of describing particular embodiments and should not be construed as limiting the scope of the claims in any way.

[0092] Therefore, it should be understood that the above description is illustrative and not restrictive. Many embodiments and applications other than the examples provided will become apparent upon reading the above description. The scope of the disclosure should be determined not with reference to the above description, but with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technology discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In short, it should be understood that this application is capable of modification and variation.

[0093] All terms used in the claims are intended to be given their broadest reasonable construction and their ordinary meaning as understood by those skilled in the art described herein, unless expressly indicated to the contrary. In particular, the use of singular articles such as "a," "the," "said," etc., should be read as describing one or more of the indicated elements, unless an express to the contrary limitation is recited in the claims.

[0094] The Abstract of the present disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Additionally, in the above Detailed Description, various features may be seen grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments include more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Accordingly, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as separately claimed subject matter.

[0095] While this disclosure includes reference to exemplary embodiments, this specification is not intended to be construed in a limiting sense. Various modifications of the described embodiments that are apparent to those skilled in the art to which this disclosure pertains, as well as other embodiments within the scope of this disclosure, are deemed to be within the principles and scope of the disclosure, as expressed, for example, in the following claims.

[0096] Some embodiments may be implemented as circuit-based processes, including potentially implementation on a single integrated circuit.

[0097] Some embodiments may be embodied in the form of methods and apparatuses for practicing those methods. Some embodiments may also be embodied in the form of program code recorded on tangible media, such as magnetic recording media, optical recording media, solid-state memory, floppy diskettes, CD-ROMs, hard drives, or any other non-transitory machine-readable storage media. When the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the patented invention. Some embodiments may also be embodied in the form of program code stored on a non-transitory machine-readable storage medium, for example, including being loaded into and / or executed by a machine. When the program code is loaded into and executed by a machine, such as a computer or processor, the machine becomes an apparatus for practicing the patented invention. When implemented on a general-purpose processor, the program code segments combine with the processor to provide a unique device that operates analogously to specific logic circuits.

[0098] Unless expressly stated otherwise, each numerical value and range should be construed as approximate, as if the word "about" or "approximately" were before the value or range.

[0099] The use of figure numbers and / or figure reference signs in the claims is intended to identify one or more possible embodiments of the claimed subject matter to facilitate claim interpretation, and such use should not necessarily be construed as limiting the scope of those claims to the embodiments shown in the corresponding figures.

[0100] In the method claims below, elements, if any, are described in a particular sequence with corresponding labeling, but the elements are not necessarily intended to be limited to being implemented in that particular sequence, unless the claim recitation otherwise suggests a particular sequence for implementing some or all of the elements.

[0101] As used herein, "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of the present disclosure. The appearances of the phrase "in one embodiment" in various places in this specification do not necessarily all refer to the same embodiment, nor do they necessarily refer to separate or alternative embodiments that are mutually exclusive of other embodiments. The same applies to the term "implementation."

[0102] Unless otherwise specified herein, the use of ordinal adjectives "first," "second," "third," etc. to refer to multiple similar objects merely indicates that different instances of such similar objects are being referenced, and is not intended to imply that the similar objects so referenced must be in a corresponding order or sequence in time, space, precedence, or otherwise.

[0103] Unless otherwise specified herein, the conjunction "if" may be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting," in addition to its plain meaning, and these interpretations may depend on the corresponding specific context. For example, the phrase "when determined" or "when (a referenced condition) is detected" may be interpreted as "upon determining" or "in response to determining" or "upon detecting (a referenced condition or event)" or "in response to detecting (a referenced condition or event)."

[0104] Also, for purposes of this description, "couple," "coupling," "coupled," "connect," "connecting," or "connected" refers to any manner known in the art or later developed by which energy can be transferred between two or more elements, which does not necessarily involve one or more additional elements. Conversely, terms such as "directly coupled," "directly connected," and the like imply the absence of such additional elements.

[0105] As used herein with respect to elements and standards, the term compatible means that an element communicates with other elements in a manner specified in whole or in part by the standard and is recognized by other elements as being sufficiently capable of communicating with them in the manner specified by the standard. A compatible element need not operate internally in the manner specified by the standard.

[0106] The functions of the various elements shown in the figures, including any functional blocks labeled "processor" and / or "controller," may be provided through the use of dedicated hardware as well as hardware capable of executing software in conjunction with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by multiple individual processors, some of which may be shared. Furthermore, explicit use of the terms "processor" or "controller" should not be construed as referring exclusively to hardware capable of executing software, but may implicitly include digital signal processor (DSP) hardware, network processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), read-only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage. Other hardware, conventional and / or custom, may also be included. Similarly, any switches shown in the figures are conceptual only. The functions of those switches may be performed by the operation of program logic, by dedicated logic, by the interaction of program control and dedicated logic, or even manually, with the particular technique being selectable by the implementer, as more particularly understood from the context.

[0107] As used in this application, the term “circuit” may refer to one or more or all of the following: (a) a hardware-only circuit implementation (such as an implementation in only analog and / or digital circuitry); (b) a combination of hardware circuitry and software, such as (where applicable): (i) a combination of analog and / or digital hardware circuitry and software / firmware; and (ii) a hardware processor (including a digital signal processor) with software, software, and any portion of memory that cooperate to cause a device such as a cell phone or server to perform various functions; and (c) a hardware circuit and / or processor, such as a microprocessor or portion of a microprocessor, that requires software (e.g., firmware) for operation but may not be present when not required for operation. This definition of circuit applies to all uses of this term in this application, including the use of this term in any claims. As a further example, as used in this application, the term “circuit” also covers implementations solely of a hardware circuit or processor (or processors) or portion of a hardware circuit or processor, along with its (or their) accompanying software and / or firmware. The term circuitry also covers, for example, baseband or processor integrated circuits for mobile devices, or similar integrated circuits in servers, cellular network devices, or other computing or network devices, where applicable to particular claim elements.

[0108] Those skilled in the art should understand that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the present disclosure. Similarly, it will be understood that any flowcharts, flow diagrams, state transition diagrams, pseudocode, etc. may be substantially represented on a computer-readable medium and represent various processes that may be executed by a computer or processor, whether or not a computer or processor is explicitly shown.

[0109] This Summary is intended to introduce some example embodiments, with further embodiments described in the Detailed Description and / or with reference to one or more of the drawings. This Summary is not intended to identify essential elements or features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

Claims

1. 1. An image processing system for encoding spherically sampled data, comprising: The image processing system includes a processor that encodes data, the processor receiving a three-dimensional environment; For a set of cube map texture coordinates, applying a cube map function to the cube map texture coordinates to generate a cube map orientation; applying a low-discrepancy sequence sampling function to the cube map directions to generate low-discrepancy spherical sampling directions; applying an offset cube mapping function to the low discrepancy spherical sampling directions to generate offset sampling directions; and generating pixel values ​​for the cube map texture coordinates using the offset sampling direction and the three-dimensional environment; configured to: Image processing system.

2. the low discrepancy sequence sampling function is a unicube sampling function; The image processing system according to claim 1 .

3. the low discrepancy sequence sampling function is an isocube uniform sampling function; The image processing system according to claim 1 .

4. the processor is further configured to provide the pixel values ​​of the cube map texture coordinates via a virtual reality device. The image processing system according to any one of claims 1 to 3.

5. The processor: receiving a gaze vector indicating a direction of a user's line of sight; determining a rate of change of the gaze vector; selecting an offset value based on the rate of change of the line of sight vector; configured to apply the offset cube mapping function by performing operations including: The image processing system according to any one of claims 1 to 4.

6. Selecting the offset value comprises: selecting a first offset value when the rate of change of the gaze vector is less than a first threshold; and selecting a second offset value when the rate of change of the gaze vector is greater than or equal to the first threshold; Including, the first offset value is greater than the second offset value; The image processing system according to claim 5 .

7. The processor: receiving a gaze vector indicating a user's line of sight; and aligning the offset sampling direction with the line of sight vector; further configured to: The image processing system according to any one of claims 1 to 4.

8. the processor is further configured to select a sampling frequency of the low discrepancy sequence sampling function. The image processing system according to any one of claims 1 to 7.

9. the processor is configured to apply the cube map function to the cube map texture coordinates in a shader; The image processing system according to any one of claims 1 to 8.

10. 1. An image processing system for encoding spherically sampled data, comprising: The image processing system includes a processor that encodes data, the processor receiving a three-dimensional environment; applying an offset low discrepancy sequence (OLDS) sampling function to the set of cube map texture coordinates to generate an OLDS direction; and generating pixel values ​​for the cube map texture coordinates using the OLDS orientation and the three-dimensional environment; configured to: Image processing system.

11. The OLDS sampling function includes a unicube sampling function. The image processing system according to claim 10.

12. the OLDS sampling function comprises an isocube uniform sampling function; The image processing system according to claim 10.

13. The processor: receiving a gaze vector indicating a direction of a user's line of sight; determining a rate of change of the gaze vector; selecting an offset value based on the rate of change of the line of sight vector; and applying an offset cube mapping function using the offset values; configured to apply the OLDS sampling function by performing operations including: The image processing system according to any one of claims 10 to 12.

14. Selecting the offset value comprises: selecting a first offset value when the rate of change of the gaze vector is less than a first threshold; and selecting a second offset value when the rate of change of the gaze vector is greater than or equal to the first threshold; Including, the first offset value is greater than the second offset value; The image processing system according to claim 13.

15. The processor: receiving a gaze vector indicating a user's line of sight; and aligning the offset sampling direction of the OLDS sampling function with the line of sight vector; further configured to: The image processing system according to any one of claims 10 to 14.

16. The processor: further configured to select a sampling frequency of the OLDS sampling function. The image processing system according to any one of claims 10 to 15.

17. 1. An image processing system for decoding spherically sampled data, the image processing system comprising: a processor for decoding the data, the processor comprising: receiving an offset-sampled cube map from an encoded bitstream; for an offset sampling direction, applying an inverse offset cube mapping function to said offset sampling direction to generate a low discrepancy spherical sampling direction; applying an inverse low discrepancy sequence sampling function to the low discrepancy spherical sampling directions to generate cube-map directions; applying an inverse cube map function to the cube map orientation to generate cube map texture coordinates; and generating pixel values ​​in the offset sampling direction using the cube map texture coordinates and the offset sampled cube map; configured to: Image processing system.

18. the inverse low discrepancy sequence sampling function is an inverse unicube sampling function; 18. The image processing system according to claim 17.

19. the inverse low discrepancy sequence sampling function is an inverse isocube sampling function; 18. The image processing system according to claim 17.

20. The processor: and further configured to receive, from the encoded bitstream, metadata associated with the offset-sampled cube-map, the metadata including the inverse offset cube-mapping function. The image processing system according to any one of claims 17 to 19.

Images