Offset low disparity spherical sampling for image rendering

Abstract

Systems and methods for 360 DEG image rendering. An example image processing system includes a processor to perform data encoding, the processor configured to receive a three-dimensional environment. The processor is configured to: for a set of cube map texture coordinates, apply a cube map function to the cube map texture coordinates to generate a cube map direction; applying the low-difference sequence sampling function to the cube mapping direction to generate a low-difference spherical sampling direction; and applying the offset cube mapping function to the low-disparity spherical sampling direction to generate an offset sampling direction. The processor is configured to generate pixel values for the cube map texture coordinates using an offset sampling direction and a three-dimensional environment.

Description

[0001] 1. Cross-reference to related applications This application claims priority to the following priority applications: U.S. Provisional Application 63 / 386,559 filed on December 8, 2022 (reference number: D22061USP1) and European Application No. 23166550.6 filed on April 4, 2023 (reference number: D22061EP), each of which is hereby incorporated herein in its entirety. 2. Technical Field Various example embodiments relate to image processing operations, and more particularly, but not exclusively, to 360° image rendering. 3. Background Technology Human visual sensitivity peaks in the fovea, where the concentration of cones is highest, and decreases with increasing retinal eccentricity. Foveated rendering refers to a range of computer graphics techniques in which the resolution of the rendered image is tailored to match the visual sensitivity of the viewer's eyes. That is, more detail is provided at the center of the viewer's gaze, while progressively less detail is provided further away from that point. This technique is being implemented in virtual reality (VR) and augmented reality (AR) applications because foveated rendering offers reduced rendering times and reduced data transmission bandwidth.

[0004] Cubemap texturing is a technique for representing three-dimensional (3D) environment maps, such as 360° still images or video frames. Specifically, a textured 3D cube is represented as six two-dimensional (2D) textures, one for each cube face. Modern graphics processing units (GPUs) provide hardware support for cubemap texturing, including handling sampling across multiple cube faces. Summary of the Invention

[0005] Various embodiments of image processing systems for encoding and decoding spherical sampling data are disclosed herein. One exemplary embodiment provides an image processing system comprising a processor for performing data encoding, the processor being configured to receive a three-dimensional environment. The processor is configured to: for a set of cubemap texture coordinates, apply a cubemap function to the cubemap texture coordinates to generate cubemap directions; apply a low-discrepancy sequence sampling function to the cubemap directions to generate low-discrepancy spherical sampling directions; and apply an offset cubemap function to the low-discrepancy spherical sampling directions to generate offset sampling directions. The processor is configured to use the offset sampling directions and the three-dimensional environment to generate pixel values ​​for the cubemap texture coordinates.

[0006] According to an example embodiment, an image processing system for encoding spherical sampling data is provided. The image processing system includes a processor for performing data encoding. The processor is configured to receive a three-dimensional environment, apply an offset low-discrepancy sequence (OLDS) sampling function to a set of cubemap texture coordinates to generate OLDS directions, and generate pixel values ​​for the cubemap texture coordinates using the OLDS directions and the three-dimensional environment.

[0007] According to another example embodiment, an image processing system for decoding spherical sampling data is provided. The image processing system includes a processor for performing data decoding, the processor being configured to receive an offset sampling cubemap from an encoded bitstream. The processor is configured to: for an offset sampling direction, apply an inverse offset cubemap function to the offset sampling direction to generate a low-disparity spherical sampling direction, apply an inverse low-disparity sequence sampling function to the low-disparity spherical sampling direction to generate a cubemap direction, and apply an inverse cubemap function to the cubemap direction to generate a cubemap texture coordinate. The processor is configured to generate a pixel value for the offset sampling direction using the cubemap texture coordinate and the offset sampling cubemap. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Other aspects, features, and advantages of the various disclosed embodiments will become more fully apparent from the following detailed description and accompanying drawings, by way of example, in which: Figure 1 Depicted is an example process of a video transmission pipeline according to one embodiment.

[0009] Figure 2 Depicted is an example representation of the field of view of an average viewer's eye.

[0010] Figure 3 Depicted is an example cubemap within a 3D environment, according to one embodiment.

[0011] Figures 4A to 4C Depicted is an example offset cubemap within a 3D environment, according to one embodiment.

[0012] Figures 5A to 5C Depicting various example sampling density uniformities for a cubemap.

[0013] Figures 6A to 6C Depicting various example sampling density uniformities for an offset cubemap.

[0014] Figure 7 Depicts the area of ​​triangles from a Delaunay triangulation of an offset standard cubemap spherical sample.

[0015] Figure 8Depicts the area of ​​triangles from a Delaunay triangulation sampled from an offset uniform cubemap sphere.

[0016] Figure 9 Depicted is a diagram of a Figure 1 Example process performed by the video transmission pipeline.

[0017] Figure 10 Depicted is a diagram of a Figure 1 Another example process performed by the video transmission pipeline. DETAILED DESCRIPTION

[0018] The present disclosure and its aspects may be embodied in various forms, including: hardware, devices or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces and application programming interfaces; as well as hardware-implemented methods, signal processing circuits, memory arrays, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc. The foregoing is intended only to give a general idea of ​​the various aspects of the present disclosure and is not intended to limit the scope of the disclosure in any way.

[0019] The video application as described herein may refer to any of the following: a video display application, a VR application, an AR application, an automotive entertainment application, a telepresence application, a display application, a gaming application, a mobile application, an Internet-based video streaming application, and the like.

[0020] These techniques can be applied to immersive movie viewing in virtual reality theaters (e.g., for head-mounted displays, etc.) to stream video data between (multiple) video streaming servers and (multiple) video streaming clients. Example video content may include, but is not necessarily limited to, any of the following: audiovisual programs, movies, video programs, TV broadcasts, computer games, AR content, VR content, automotive entertainment content, etc. Example video streaming clients may include, but are not necessarily limited to, any of the following: display devices, computing devices with near-eye displays, head-mounted displays (HMDs), mobile devices, wearable display devices, set-top boxes with displays such as televisions, video monitors, etc.

[0021] As used herein, a "video streaming server" may refer to one or more upstream devices that prepare video content and stream the video content to one or more video streaming clients so that at least a portion of the video content (e.g., a portion corresponding to a user's FOV or viewport, etc.) is rendered on one or more (target) displays. The display on which the video content is rendered may be part of one or more video streaming clients, or may operate in conjunction with one or more video streaming clients. Example video streaming servers may include, but are not necessarily limited to, any of the following: a cloud-based video streaming server located remotely from the video streaming client(s), a local video streaming server connected to the video streaming client(s) via a local wired or wireless network, a VR device, an AR device, an automotive entertainment device, a digital media device, a digital media receiver, a set-top box, a game console, a general-purpose personal computer, a tablet computer, a dedicated digital media receiver, and the like.

[0022] Video Encoding According to Example Embodiments Figure 1 An example process of a video transmission pipeline 100 according to an embodiment is depicted, showing various stages from video capture to video content display. An image generation block 105 can be used to capture or generate a sequence of video frames 102. Video frames 102 can be digitally captured (e.g., by a digital camera) or generated by a computer (e.g., using computer animation) to provide video data 107. Alternatively, video frames 102 can be captured on film by a film camera. The film can then be converted to a digital format to provide video data 107.

[0023] During the production phase 110, video data 107 may be edited to provide a video production stream 112. The data from video production stream 112 may then be provided to a processor (or one or more processors, such as a central processing unit (CPU)) at a post-production block 115 for post-production editing. Post-production editing at block 115 may include, for example, adjusting or modifying the color or brightness in specific areas of an image to enhance image quality or achieve a specific look based on the video creator's creative intent. This portion of post-production editing is sometimes referred to as "color timing" or "color grading." Other editing may be performed at block 115 (e.g., scene selection and sequencing, image cropping, adding computer-generated visual effects, etc.) to produce a "final" version 117 of the production for release. For example, rendering techniques such as cube mapping, cubemap offsetting, and sampling, as described herein, may be performed during post-production editing 115. During post-production editing 115, the video image may be viewed on a reference display 125. In other examples, the rendering techniques described herein may be performed at the image generation block 105 , in the production stage 110 , or at another suitable processing step within the video transmission pipeline 100 .

[0024] After post-production 115, the video data of the final version 117 can be transmitted to an encoding block 120 for downstream transmission to decoding and playback devices such as VR headsets, near-eye displays, and the like. In some embodiments, encoding block 120 may include audio and video encoders (such as those defined by ATSC, DVB, DVD, Blu-ray, and other transmission formats) to generate an encoded bitstream 122. In the receiver, encoded bitstream 122 is decoded by a decoding unit 130 to generate a corresponding decoded signal 132 representing a replica or a close approximation of signal 117. The receiver may be attached to a target display 140, which may have slightly different or completely different characteristics than the reference display 125. In this case, a display management (DM) block 135 may be used to map the decoded signal 132 to the characteristics of the target display 140 by generating a display mapping signal 137. Depending on the embodiment, decoding unit 130 and display management block 135 may include separate processors or be based on a single integrated processing unit. The encoded bitstream 122 may include metadata to assist in reconstructing the final version 117.

[0025] Human Vision Techniques as described herein may be implemented based on knowledge of the (human) eye and how it perceives brightness, color, spatial resolution, etc. These techniques may be applied based on the specific visual characteristics of an individual viewer or the average or collective visual characteristics of a group of users.

[0026] Figure 2 An example representation of the field of view of an average viewer's eye is shown. Figure 2 As shown, the distribution of cones and rods (in the eye) can be segmented into different cone distribution ranges and rod distribution ranges, and can be further projected into an angular field of view (of the eye). The highest level of visual perception is achieved in the fovea (field of view) region 202 of the eye.

[0027] By way of example and not limitation, the widest angular range in the eye's field of view is along Figure 2 The horizontal direction is parallel to the interpupillary line between the viewer's two eyes without considering the visual constraints from the facial anatomy and can be approximately 180 degrees.

[0028] Each concentric circle (e.g., labeled 30°, 60°, 90°, etc.) represents a direction with an equal (or identical) angle relative to the viewing direction of the viewer's left or right eye. It should be noted that angles such as 30°, 60°, 90°, etc. are for illustration purposes only. Different angle values ​​or different groups of angles may be used to define or describe the viewer's field of view. Viewing direction ( Figure 2 (not shown) in the central concave region 202 (the darkest fill pattern), the intersection of the lateral direction 212 and the vertical direction 214 is directed vertically Figure 2 Here, the lateral direction 212 and the vertical direction 214 form a plane orthogonal to the viewing direction.

[0029] like Figure 2 As illustrated, the field of view of the eye can be divided (e.g., logically divided, projected by certain partitions in the rod / cone density distribution) into a foveal region 202, which is closely surrounded by a paracentral region 204. In some embodiments, the foveal region 202 can correspond to the viewer's foveal vision and extend from a zero (0) angle to a first angle (e.g., 2 to 4 degrees, 3 to 7 degrees, 5 to 9 degrees, etc.) relative to the viewing direction. In some embodiments, the paracentral region 204 can extend from the first angle to a second angle (e.g., 6 to 12 degrees, etc.) relative to the viewing direction.

[0030] Paracentral region 204 is closely surrounded by near peripheral region 206. Near peripheral region 206 is adjacent to mid-peripheral region 208, which in turn is adjacent to the remainder of the field of view, namely, far peripheral region 210. In some embodiments, near peripheral region 206 may extend from a second angle to a third angle (e.g., 25 to 35 degrees, etc.) relative to the viewing direction. In some embodiments, mid-peripheral region 208 may extend from a third angle to a fourth angle (e.g., 50 to 65 degrees, etc.) relative to the viewing direction. Far peripheral region 210 may extend from the fourth angle to the edge of the field of view.

[0031] The first angle, second angle, third angle, and fourth angle used in this example logical partitioning of the field of view may be defined or specified along the lateral direction 212. Figure 2 When the field of view corresponds to the field of view in the front horizontal viewing direction, the lateral direction 212 can be the same as or parallel to the viewer's interpupillary line.

[0032] It should be noted that additionally or instead of based on Figure 2 The angles shown logically divide the viewer's field of view into a foveal area, a paracentral area, a near peripheral area, a mid-peripheral area, and a far peripheral area. Different schemes for logically dividing the viewer's field of view may be used.

[0033] For example, in some embodiments, the viewer's field of view may be divided into more or fewer regions, such as a combination of a foveal region, a near peripheral region, and a far peripheral region, without a paracentral region and / or a mid-peripheral region. Spatially faithfully represented (or high-fidelity) image portions may be used to cover some or all of such logical partitions of the viewer's field of view, from the foveal region to the near peripheral region.

[0034] In some embodiments, the Figure 2 The viewer's field of view may be divided by other measures other than the angles illustrated. For example, in a non-limiting embodiment, the foveal region may be defined as the region of the visual field corresponding to the viewer's foveal vision. The paracentral region may be defined as the region of the visual field corresponding to the viewer's retinal area where the cone / rod density exceeds a relatively high cone / rod density threshold. The near peripheral region may be defined as the region of the visual field corresponding to the viewer's retinal area where the cone / rod density does not exceed a relatively high cone / rod density threshold, but does exceed a medium cone / rod density threshold. The mid-peripheral region may be defined as the region of the visual field corresponding to the viewer's retinal area where the cone / rod density does not exceed a medium cone / rod density threshold, but does exceed a relatively low cone / rod density threshold. The focal visual regions described herein may cover some or all of the viewer's foveal vision up to a certain region (e.g., some or all of the viewer's near peripheral vision, etc.) based on threshold(s) that are not necessarily based on angle (e.g., cone / rod density threshold(s), etc.).

[0035] Additionally, optionally or alternatively, a combination of two or more different schemes for logically partitioning the viewer's field of view and / or other human vision factors can be used to determine the focal visual area of ​​the viewer's field of view. For example, instead of using the focal visual area as described herein to cover the same range of angular values ​​in different angular directions, the focal visual area as described herein can cover a larger range of angular values ​​along the lateral direction 212 than the range of angular values ​​covered by the focal visual area along the vertical direction 214 because the human visual system is more sensitive to image details along the lateral direction 212 than to image details along the vertical direction 214.

[0036] In some embodiments, the focal visual area as described herein covers some or all of the following areas: the foveal area (e.g., plus a safety margin, etc.), the paracentral area (e.g., excluding the foveal area and extending from the foveal area, etc.), the near peripheral area (e.g., further excluding the paracentral area and extending from the paracentral area, etc.), the mid-peripheral area (e.g., further excluding the near peripheral area and extending from the near peripheral area, etc.), etc.

[0037] Additionally, optionally or alternatively, in some embodiments, the eye field of view as described herein takes into account vision-related factors such as eye rotation, viewing constraints from the nose, cornea, eyelids, etc.

[0038] Examples of focal visual areas as described herein may include, but are not necessarily limited to, any combination of one or more of the following: a perfect circular shape, a rectangular shape, an oval shape, a heart shape, a star shape, a circular shape, a square shape, a polygonal shape, etc.

[0039] Gaze Tracking In some embodiments, only a (relatively small) focal visual area of ​​the eye's field of view need be configured with pixel values ​​having the highest dynamic range, widest color gamut, highest (or sharpest) spatial resolution, etc. In some embodiments, the focal visual area of ​​the eye's field of view may approximately correspond to the entire foveal vision of the eye through some or all of the eye's near peripheral vision. In some embodiments, the focal visual area of ​​the eye's field of view may additionally include a safety field of view area.

[0040] In some embodiments, the size and / or shape of the safe field of view area within the focal visual area can be preconfigured to a fixed size (e.g., 0%, 5%, 10%, -5%, -10%, etc.) that does not vary with network bandwidth, image content, the type of computing device involved in the video application (e.g., a helmet-mounted display device, a wall display, etc.), the type of rendering environment involved in the video application (e.g., a cloud-based video streaming server, etc.), etc. In some other embodiments, the size and / or shape of the safe field of view area within the focal visual area can be dynamically reconfigured at runtime and can vary within a certain range. For example, in response to determining that the network connection does not support relatively high bandwidth, the size and / or shape of the safe field of view area can be dynamically reduced from 10% to 5% relative to the eye's foveal vision at runtime. On the other hand, in response to determining that the network connection does support relatively high bandwidth, the size and / or shape of the safe field of view area can be dynamically expanded from 5% to 10% relative to the eye's foveal vision at runtime.

[0041] The size and / or shape of the safe field of view area can also be set based on the latency in eye tracking. Additionally, optionally or alternatively, eye tracking data as described herein can be used to predict where a viewer will be looking, and bandwidth / safe areas can be reduced based on this prediction. For example, a user's viewing direction at runtime can be tracked by a viewing direction tracking device. The viewing direction tracking device can operate in real time with a display on which the image sequence of the display map is rendered. As the user changes viewing direction and / or viewing distance from time to time, the viewing direction tracking device tracks and calculates the viewing angle and / or viewing distance in the coordinate system in which the image sequence of the display map is being rendered, generates a time series of viewing directions, and signals each viewing direction in the time series of viewing directions to a video streaming server as described herein. Each such signaled viewing direction of a viewer received by the video streaming server can be indexed by a time point value. The time point value can be associated or correlated with an offset LDSS cubemap by the video streaming server as described herein.

[0042] While the viewer is viewing the 3D environment or a derivative thereof, viewing direction data is collected to track the viewer's viewing direction. Example viewing direction data may include, but is not limited to, linear displacement, angular displacement, linear motion or translation, angular motion or rotation, pitch, roll, yaw, roll, heave, surge, etc., which may be collected by any combination of gaze tracking devices, facial tracking devices, FOV tracking devices, etc. The viewer's viewing direction may be determined at multiple points in time. The viewer's viewing direction may be used on the receiving device itself to generate a new view by implementing some or all of the techniques described herein on the receiving device (or a device operating in conjunction with the receiving device) until the upstream device responds with new data.

[0043] The viewing direction data can be collected, analyzed, and / or shared / transmitted between the viewing direction tracking device and the streaming device with relatively low latency (e.g., within a fraction of an image frame time, within 5 milliseconds, etc.). In an example embodiment, where multiple data / network connections are available, the viewing direction tracking data can be shared between these devices using the lowest latency data / network connection. The viewing direction data may be referred to herein as a gaze vector.

[0044] In response to relatively rapid determination of the viewer's viewing direction based on the viewing direction data with relatively low latency (e.g., within a fraction of an image frame time), the video streaming server can dynamically reduce the size and / or shape of the safe field of view area from 10% to 5% relative to the eye's foveal vision at runtime. A relatively small area (e.g., within 20 degrees from the viewing direction, etc.) with the highest dynamic range, widest color gamut, highest spatial resolution, etc. can be sent to downstream receiving devices in the video signal.

[0045] On the other hand, in response to a relatively slow determination of the viewer's viewing direction based on the viewing direction data with relatively high latency (e.g., exceeding a certain time threshold, exceeding a fraction of an image frame time, exceeding 5 milliseconds, etc.), the video streaming server can dynamically expand the size and / or shape of the safe field of view area relative to the eye's foveal vision at runtime from 1% to 3%, from 2% to 6%, from 5% to 10%, etc. A relatively large area (e.g., up to 30 degrees from the viewing direction, etc.) with the highest dynamic range, widest color gamut, highest spatial resolution, etc. can be sent to the downstream receiving device in the video signal. In this way, the receiving device of the video signal can have sufficient image data over the relatively large focal visual area to make local decisions based on the viewing direction for image rendering purposes.

[0046] Cube Mapping Technology The embodiments described herein relate to a 3D virtual environment captured using one or more cameras or generated by a computer graphics engine. The 3D virtual environment is then accessed using an immersive device, such as a virtual reality headset. To assist in rendering an image where the resolution is maximized at the center of the viewer's gaze, a foveated rendering technique can be implemented. One technique for representing a 3D environment using one or more 2D textures involves mapping the 3D virtual environment onto a cube, thereby generating a cubemap. As an example, in a computer graphics program operating on the instantaneous state of a 3D model graph (e.g., a scene), six virtual cameras with a common focal point are constructed. These six cameras observe along the positive and negative directions of three orthogonal axes, such that each camera is pointed toward the center of a face of the cube. This common focal point can be located anywhere within the scene, and the six-camera arrangement can be arbitrarily oriented within the scene.

[0047] Figure 3 Provides example cube mapping techniques for 3D environments (e.g., scenes). Figure 3 In the example of FIG, 3D environment 300 includes a textured cubemap 305. The center of cubemap 305 is located at an arbitrary point within 3D environment 300. 3D environment 300 is mapped onto cubemap 305. For example, cubemap 305 includes a first face 315. A camera perspective 310 (e.g., a virtual camera) is located at the center of cubemap 305 and faces outward perpendicularly from the center of first face 315. The portion of 3D environment 300 captured by camera perspective 310 is mapped onto first face 315. This process is repeated for each face of cubemap 305, so that the entire 3D environment 300 is mapped onto cubemap 305.

[0048] For the example six-camera arrangement, a virtual cube (e.g., cubemap 305) encloses the six-camera arrangement, with the center of cubemap 305 coinciding with the focal points of the cameras. The focal points of the six cameras can be anywhere within the scene, not necessarily at the center. Three orthogonal axes pass through the centers of faces (e.g., first face 315) of cubemap 305. Uniform sampling of each face of the virtual cube produces six images, which form the textured cubemap 305. Each image is a view of the scene captured by one of the six cameras. In some examples, each camera has a field of view of 90 degrees.

[0049] However, cube mapping can cause spherical samples (or 3D model maps) to be more heavily concentrated near the spherical projections of the cubemap's edges and corners, and less concentrated near the projections at the cube's face centers. This results in higher resolution (in pixels per solid angle) at the corners and edges of the cube, and lower resolution at the center of each face (relative to the edges and corners). When the cubemap is rendered in the viewport, this non-uniformity causes area and angle distortions and artifacts in the environment map. Additionally, when rendering video, standard cubemap sampling wastes rendering time and bandwidth for directions away from the viewer's gaze.

[0050] To address this issue, an offset cubemap technique can be implemented. Offset cubemap is a foveated rendering technique in which the virtual camera used to generate the six faces of the cubemap texture is offset or shifted along an axis. This offset increases detail in the offset direction and decreases detail in the opposite direction. For example, Figures 4A to 4C Provides an example offset cube mapping technique for 3D environments. Figure 4A A 3D environment 400 is illustrated, comprising a first virtual camera 405, a textured cubemap 412 comprising six faces, Figure 4A Four of the faces (front face 410 , back face 415 , and side faces 420 ) are illustrated in FIG. Figure 4B 4 illustrates a second virtual camera 425 offset by an offset distance 427 (eg, an offset offset) and a textured offset cubemap 432 comprising six faces in a 3D environment 400. Figure 4B Four of the faces (front face 430, back face 435, and side faces 440) are shown in FIG. Offset distance 427 may be a vector having a magnitude and a direction away from the common center of textured cubemap 412 and textured offset cubemap 432. The common center of textured cubemap 412 and textured offset cubemap 432 is located at a common point within 3D environment 400. By offsetting second virtual camera 425, second virtual camera 425 is closer to the edge of 3D environment 400 than first virtual camera 405. The first camera perspective maps a portion of the 3D environment onto front face 410 of textured cubemap 412, and the offset camera perspective maps a portion of 3D environment 400 onto front face 430 of textured offset cubemap 432.

[0051] Figure 4CThe illustration shows that the second virtual camera 425 captures a narrower view 445 of the environment 400 onto the front side 430 (as depicted by the smaller angular range subtended by 430 relative to 410) than the first virtual camera 405, thus providing a higher level of detail in the front direction. Similarly, the offset back-facing camera (not shown) captures a wider view 450 of the environment 400 onto the back side 435 than the back-facing standard cubemap camera (not shown), thus providing a lower level of detail in that direction.

[0052] For the six-camera arrangement example, a virtual cube (e.g., first textured cubemap 405) encloses the six-camera arrangement, as previously described. However, the cube center is offset from the common camera focal point along one of the orthogonal axes. As a result, each camera's view of the scene is different, with some cameras having a narrower view of the scene and some cameras having a wider view of the scene. When each camera samples its corresponding cube face at the same sampling density, cameras with a narrower view of the environment 400 sample the scene at a higher density than cameras with a wider view.

[0053] Functionally, offset cubemaps are performed by applying a cubemap texture coordinate transform applied in the shader before executing the cubemap sampling function. By reducing the number of pixels rendered in directions away from the viewer's look vector, the offset cubemap reduces both rendering time and required bandwidth. However, offset cubemaps may experience spatial sample distortion due to the offset.

[0054] Low-discrepancy spherical sampling The embodiments described herein provide for applying a virtual camera bias of an offset cubemap to low-disparity spherical sampling (LDSS). Disparity as described herein refers to a measure of the equidistribution of a set of points on a sphere, where lower disparity values ​​mean more uniform sampling. Further details on the equidistribution of a set of points on a sphere can be found in "J. Cui et al., Equidistribution on the Sphere, 18, No. 2: pp. 595-609, 1997, which is incorporated herein by reference. The LDSS technique described herein exhibits lower disparity, area, and angular distortion than a standard (e.g., unbiased) cubemap (such as the cubemap shown in FIG5 ).

[0055] An example LDSS technique is equal cube sampling. Equal cube sampling involves first dividing the spherical surface into four equatorial regions and two polar regions, with each equatorial region and each polar region having equal areas. Each region is then divided into N × N elements. When the generated N × N When elements are mapped to a cube, these elements are relatively curved (e.g., twisted) compared to samples from a standard cubemap. Samples on an isocubemap are indexed on a ring parallel to the equator of the sampled sphere. Further details on isocube sampling can be found in "L. Wan et al., Isocube Exploiting the Cube map Hardware body], 13, No. 4: pp. 720–731, July / August 2007, which is incorporated herein by reference.

[0056] Another example LDSS technique is uniform cube sampling. Uniform cube sampling involves partitioning the cube face based on two sets of parallel grid lines. These parallel lines are non-uniformly distributed so that the partitions on the projected spherical surface maintain a uniform rectilinear structure. The non-uniform partitions of the cube face are then mapped to uniform partitions. Further details on uniform cube sampling can be found in "Tze-Yiu Ho et al., Unicube for Dynamic Environment Mapping uniform cube mapped to the dynamic environment], 17, No. 1: pp. 51–63, January 2011, which is incorporated herein by reference.

[0057] and Figure 3 Compared to standard cube mapping, the LDSS technique provides improved sampling density uniformity. Figure 5A A uniform cubic cubemap 500 is illustrated having a uniform cubic sampling density. Figure 5B A standard cubemap 510 is illustrated with a non-uniform (eg, no LDSS technique applied) sampling density. Figure 5C An isocubic cubemap 520 is illustrated having an isocubic sampling density. Compared to the standard cubemap 510, the uniform cubic cubemap 500 and the isocubic cubemap 520 are more uniformly sampled.

[0058] In the same way that the LDSS technique provides improved sample distribution for standard cubemap sampling, the LDSS technique also improves the sample distribution for offset cubemaps (Offset LDSS Cubemaps). For example, Figure 6A A spherical projection of an offset uniform cubic cubemap 600 is illustrated with both a uniform cubic sampling density and an offset bias. Figure 6BA spherical projection of an offset standard cubemap 610 is illustrated with both the standard cubemap sampling density and an offset bias. Figure 6C The spherical projection of an offset isocubic cubemap 620 with both isocubic sampling density and offset bias is shown. Compared to offset cubemap 610, offset uniform cubemap 600 and offset isocubic cubemap 620 are more uniformly sampled (ignoring the expected spatial distortion of sample coordinates due to the offset). Thus, implementing both the offset bias and LDSS techniques improves the resulting spatial sample distribution relative to using only the offset bias. Furthermore, implementing both the offset bias and LDSS techniques results in better image quality, reduced transmission bandwidth requirements, and (in some instances) better video compression ratios.

[0059] As a further example of the benefits of offsetting LDSS cubemaps, Figure 7 Figure 2 shows the triangle areas from the Delaunay triangulation of an offset cubemap without LDSS applied. Figure 7 As shown, triangles with smaller areas (and therefore higher resolution) are positioned along the spherical projection regions corresponding to the edges and corners of the cubemap faces. For comparison, Figure 8 Illustration of the triangle areas from the Delaunay triangulation of an offset uniform cubemap. Figure 7 compared to, Figure 8 The area of ​​the triangle in decreases monotonically as it approaches the center of the offset surface.

[0060] Referring to the six-camera arrangement example and Figure 4, before the cubemap offset, the cameras had a relatively uneven sampling density. After implementing the LDSS technique, the cameras have a more uniform sampling density. When the sample directions are projected onto the surfaces of concentric spheres, the sample positions are shifted to provide a more uniform sample distribution.

[0061] Both offset cube mapping and LDSS techniques can be thought of as functions that take as input a sample direction (expressed as a 3D Cartesian coordinate or a 2D spherical angle) and produce a modified direction as output. For example, let x = (x, y, z) be a unit length direction vector. A direction vector, as used herein, refers to a 3-element vector within a 3D environment. A direction vector may be simply referred to as a direction (e.g., a unit length direction, a sampling direction, etc.). The equivalent or corresponding spherical angle can be calculated based on x. Let u = (u, v, f) be a cube map texture coordinate, where (u, v) are the horizontal and vertical texture coordinates, and f is a face index in the range [1, 6]. Let cm (u) =x, then the inverse relationship is expressed as u= cm -1(x), which defines the correspondence between the direction vector and the standard cubemap texture coordinate.

[0062] Let d be an axis-aligned offset direction of the form (d, 0, 0), (0, d, 0), or (0, 0, d), where d is a scalar value in the range (-1, 1). ocm (x, d) takes the direction vector x and the offset d and returns the modified direction vector x o When applied to cubemap texture coordinates, the modified offset cubemap sample direction is x o (u, d) = ocm ( cm (u),d), and the offset cubemap pixel at u (i.e., the pixel at position (u, v) on face f) is the direction x o Environment samples on .

[0063] Similarly, low-discrepancy sequence sampling is a direction modification function where x L = lds (x), where lds () denotes a uniform cubic, equicubic, or another LDSS method. Both LDSS and offset direction modifiers can be applied via function synthesis to obtain an offset low-discrepancy sequence (OLDS) sampling function x C = olds (u, d) = ocm ( lds ( cm (u),d)), which indicates that the pixel at u in the OLDS texture is oriented in the given offset d. ocm ( lds ( cm The OLDS texture map is obtained by sampling the scene at a set of discrete u values. This OLDS texture can be encoded and transmitted to the receiver. The offset d is also transmitted to the receiver, or a value agreed upon by both the sender and receiver is used.

[0064] When a receiver renders a scene for display in, for example, an AR or VR headset, the receiver obtains scene samples in multiple directions. To obtain a scene sample in direction x* from the received OLDS texture, the OLDS texture coordinate u* is determined by solving equation (1) for u*: Equation (1) To solve equation (1) for u*, apply lds (), ocm () and cm() inverse function, where these inverse functions are represented by superscript -1. First, ocm -1 () is applied to both sides of (1) to undo the offset mapping, resulting in the LDSS direction:

[0065] Next, lds -1 () applied to both sides to get the standard cubemap orientation:

[0066] Finally, cm -1 () applied to both sides to undo the cubemap orientation and get the cubemap texture coordinate u*:

[0067] Figure 9 A method 900 for generating pixel values ​​that offset a sampling direction is illustrated. The method 900 may be performed by an electronic processor included in the post-production block 115, an electronic processor included in the image generation block 105, an electronic processor included in the production block 110, or the like. At step 902, the electronic processor receives a 3D environment. The 3D environment may be, for example, a 3D environment map, a 3D game engine scene, a ray tracing environment (e.g., a ray tracer), a texture-mapped sphere, or other virtual 3D environments that can be mapped onto a cubemap.

[0068] At step 904, the electronic processor applies a cubemap function to a set of cubemap texture coordinates to generate cubemap directions. For example, for cubemap texture coordinates u=(u, v, f), the electronic processor applies the cubemap function cm () to get the cubemap direction x= cm (u).

[0069] At step 906, the electronic processor applies the low-discrepancy sequence sampling function to the cubemap directions to generate low-discrepancy spherical sampling directions. For example, the electronic processor applies the low-discrepancy sequence sampling function to the cubemap directions to generate low-discrepancy spherical sampling directions. lds () applied to cubemap direction x to obtain low-discrepancy spherical sampling direction x L = lds (x). At step 908, the electronic processor applies the offset cube mapping function to the low-discrepancy spherical sampling directions to generate offset sampling directions. For example, the electronic processor applies the offset cube mapping function ocm () is applied to the low-discrepancy spherical sampling direction x L(Given the offset d) to obtain the offset sampling direction x o = ocm (x L ,d).

[0070] At step 910, the electronic processor generates pixel values ​​for cubemap texture coordinates using the offset sampling direction and the 3D environment. For example, applying the cubemap function cm (), low-discrepancy sequence sampling function lds () and offset cube map functions ocm () generates x C = olds (u, d) = ocm ( lds ( cm (u),d)), which indicates that the pixel at u in the OLDS texture is oriented in the given offset d. ocm ( lds ( cm In some instances, pixel values ​​and / or OLDS textures are encoded and transmitted to the receiver.

[0071] Figure 10 A method 1000 for decoding an OLDS texture (e.g., an offset sampling cubemap) is illustrated. The method 1000 may be performed by an electronic processor included in the decoding block 130. At step 1002, the electronic processor receives an offset sampling cubemap. In some examples, the decoding block 130 decodes the encoded bitstream 122 to obtain the offset sampling cubemap.

[0072] At step 1004, the electronic processor applies an inverse offset cubemap function to the offset sampling cubemap to generate low-discrepancy spherical sampling directions for the offset sampling directions. For example, the encoded bitstream 122 may include metadata for assisting in reconstructing the 3D environment. In some instances, the metadata includes a definition of ocm -1 () the distance and direction of the offset distance 427. As previously described, the electronic processor ocm -1 () is applied to offset sampling direction x C To get x L to reverse the distance and direction of the offset distance 427, thereby canceling the offset distance 427.

[0073] At step 1006, the electronic processor applies the inverse low-discrepancy sequence sampling function to the low-discrepancy spherical sampling directions to generate cubemap directions. For example, as previously described, the electronic processor applies lds -1 () is applied to the inverse low-discrepancy sequence sampling direction x LTo obtain the standard cubemap direction x. At step 1008, the electronic processor applies the inverse cubemap function to the cubemap direction to generate a cubemap texture coordinate. For example, the electronic processor applies cm -1 () is applied to the cubemap direction x to obtain the cubemap texture coordinate u. At step 1010, the electronic processor uses the cubemap texture coordinate and the offset sample cubemap to generate a pixel value for the offset sample direction.

[0074] For interactive content such as VR and AR, the bias vector (d) (e.g., magnitude and / or direction) can be dynamically aligned with the gaze vector of the viewer's viewing direction to maximize rendered image quality. Furthermore, the magnitude of the bias vector can vary based on the rate of change of the gaze vector. For example, when the viewing direction moves rapidly, the magnitude of the bias vector can decrease, ensuring more consistent visual quality across rapidly changing viewing angles. In one example of a rapidly moving viewing direction, the virtual cameras used to render the left-eye and right-eye images of a head-mounted display rapidly change their orientation in response to rapidly changing user events, such as head turns, controller manipulations, and gaze direction changes. In some embodiments, the magnitude of the bias vector (e.g., offset) is determined based on a linear function. In other embodiments, the magnitude of the bias vector increases as the rate of change of the gaze vector exceeds a threshold. For example, when the rate of change is below a first threshold, a first bias vector magnitude is selected. When the rate of change is above the first threshold but below a second threshold, a second bias vector magnitude is selected. When the rate of change is above the second threshold, a third bias vector magnitude is selected.

[0075] Additionally, for linear cinematic 360° content, the magnitude of the bias vector can be varied based on the likelihood of the viewer looking in a particular direction or the importance of the action in a particular orientation. For example, for video content that includes a storyline, the viewer is likely to look in a particular direction of the story. The bias axis (e.g., the direction of the bias vector) can be aligned with these directions, and increasing the magnitude of the bias vector in that direction improves the visual quality of important areas.

[0076] While the embodiments described herein primarily relate to a full 360° environment, in some instances, less than a full 360° environment may be rendered. For example, a subset of six cube faces (e.g., five faces) may be rendered. As another example, a sub-region within a face may be rendered.

[0077] The above video transmission system and method can provide image processing and 360° image rendering. The system, method and device according to the present disclosure can adopt any one or more of the following configurations.

[0078] (1) An image processing system for encoding spherical sampling data, the image processing system comprising: a processor for performing data encoding, the processor being configured to: receive a three-dimensional environment; for a set of cube map texture coordinates, apply a cube map function to the cube map texture coordinates to generate cube map directions; apply a low-discrepancy sequence sampling function to the cube map directions to generate low-discrepancy spherical sampling directions; apply an offset cube map function to the low-discrepancy spherical sampling directions to generate offset sampling directions; and generate pixel values ​​for the cube map texture coordinates using the offset sampling directions and the three-dimensional environment.

[0079] (2) The image processing system according to (1), wherein the low-discrepancy spherical sampling operation is a uniform cubic sampling operation.

[0080] (3) The image processing system according to (1), wherein the low-discrepancy spherical sampling operation is an equicubic sampling operation.

[0081] (4) The image processing system according to any one of (1) to (3), wherein the processor is further configured to: provide the pixel value of the cube map texture coordinate via a virtual reality device.

[0082] (5) An image processing system according to any one of (1) to (4), wherein the processor is configured to apply the offset cube mapping function by performing operations including: receiving a gaze vector indicating a viewing direction of a user; determining the rate of change of the gaze vector; and selecting an offset value based on the rate of change of the gaze vector.

[0083] (6) The image processing system according to (5), wherein selecting the offset value comprises: selecting a first offset value when the rate of change of the gaze vector is less than a first threshold; and selecting a second offset value when the rate of change of the gaze vector is greater than or equal to the first threshold, wherein the first offset value is greater than the second offset value.

[0084] (7) The image processing system according to any one of (1) to (6), wherein the processor is further configured to: receive a gaze vector indicating a viewing direction of a user; and align the offset sampling direction with the gaze vector.

[0085] (8) The image processing system according to any one of (1) to (7), wherein the processor is further configured to: select a sampling frequency of the low-discrepancy sequence sampling function.

[0086] (9) The image processing system of any one of (1) to (8), wherein the processor is configured to apply the cube map function to the cube map texture coordinates in a shader.

[0087] (10) An image processing system for encoding spherical sampling data, the image processing system comprising: a processor for performing data encoding, the processor being configured to: receive a three-dimensional environment; apply an offset low-discrepancy sequence (OLDS) sampling function to a set of cube map texture coordinates to generate OLDS directions; and use the OLDS directions and the three-dimensional environment to generate pixel values ​​for the cube map texture coordinates.

[0088] (11) The image processing system according to (10), wherein the OLDS sampling function includes a uniform cubic sampling function.

[0089] (12) The image processing system according to (10), wherein the OLDS sampling function comprises an equicubic uniform sampling function.

[0090] (13) An image processing system according to any one of (10) to (12), wherein the processor applies the OLDS sampling function by performing operations including: receiving a gaze vector indicating a viewing direction of a user; determining a rate of change of the gaze vector; selecting an offset value based on the rate of change of the gaze vector; and applying an offset cube mapping function using the offset value.

[0091] (14) The image processing system of (13), wherein selecting the offset value comprises: selecting a first offset value when the rate of change of the gaze vector is less than a first threshold; and selecting a second offset value when the rate of change of the gaze vector is greater than or equal to the first threshold, wherein the first offset value is greater than the second offset value.

[0092] (15) An image processing system according to any one of (10) to (14), wherein the processor is further configured to: receive a gaze vector indicating a viewing direction of a user; and align the offset sampling direction of the OLDS sampling function with the gaze vector.

[0093] (16) The image processing system according to any one of (10) to (15), wherein the processor is further configured to: select a sampling frequency of the OLDS sampling function.

[0094] (17) An image processing system for decoding spherical sampling data, the image processing system comprising: a processor for performing data decoding, the processor being configured to: receive an offset sampling cube map from an encoded bit stream; for an offset sampling direction, apply an inverse offset cube mapping function to the offset sampling direction to generate a low-discrepancy spherical sampling direction; apply an inverse low-discrepancy sequence sampling function to the low-discrepancy spherical sampling direction to generate a cube map direction; apply an inverse cube map function to the cube map direction to generate a cube map texture coordinate; and generate a pixel value for the offset sampling direction using the cube map texture coordinate and the offset sampling cube map.

[0095] (18) The image processing system according to (17), wherein the inverse low-discrepancy sequence sampling function is an inverse uniform cubic sampling function.

[0096] (19) The image processing system according to (17), wherein the inverse low-discrepancy sequence sampling function is an inverse equicubic sampling function.

[0097] (20) The image processing system of any one of (17) to (19), wherein the processor is further configured to: receive metadata associated with the offset sampled cube map from the encoded bitstream, wherein the metadata includes the inverse offset cube map function.

[0098] With respect to the processes, systems, methods, heuristics, and the like described herein, it should be understood that although the steps of these processes, etc. have been described as being performed in a particular ordered order, these processes can be practiced with the described steps performed in an order different from that described herein. It should further be understood that certain steps can be performed simultaneously, other steps can be added, or certain steps described herein can be omitted. In other words, the process descriptions herein are provided for the purpose of illustrating certain embodiments and should in no way be construed as limiting the claims.

[0099] Accordingly, it should be understood that the above description is intended to be illustrative and not restrictive. Upon reading the above description, many embodiments and applications other than the examples provided will be apparent. The scope should not be determined with reference to the above description, but rather with reference to the appended claims and the full scope of equivalents to which such claims are entitled. It is anticipated and hoped that the technology discussed herein will develop in the future, and that the disclosed systems and methods will be incorporated into such future embodiments. In short, it should be understood that the present application is capable of modification and alteration.

[0100] All terms used in the claims are intended to be given their broadest reasonable constructions and ordinary meanings as understood by those skilled in the art described herein, unless an explicit indication to the contrary appears herein. In particular, use of singular articles such as "a," "the," and "said" should be understood to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.

[0101] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. This Abstract is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Additionally, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for the purpose of linking the disclosure together as a whole. The disclosed approach should not be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. On the contrary, as reflected in the appended claims, the inventive subject matter lies in less than all the features of a single disclosed embodiment. Accordingly, the appended claims are hereby incorporated into the Detailed Description, with each claim standing on its own as separately claimed subject matter.

[0102] Although the present disclosure includes references to illustrative embodiments, this description is not intended to be construed as limiting. Various modifications of the described embodiments, as well as other embodiments within the scope of the present disclosure, that are apparent to those skilled in the art to which the present disclosure relates are deemed to be within the principles and scope of the present disclosure, for example, as expressed in the appended claims.

[0103] Some embodiments may be implemented as a circuit-based process, including possible implementation on a single integrated circuit.

[0104] Some embodiments may be embodied in the form of methods and apparatus for practicing these methods. Some embodiments may also be embodied in the form of program code recorded on a tangible medium, such as a magnetic recording medium, an optical recording medium, a solid-state memory, a floppy disk, a CD-ROM, a hard drive, or any other non-transitory machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine (e.g., a computer), the machine becomes an apparatus for practicing the patented invention(s). Some embodiments may also be embodied in the form of program code, for example, stored on (including being loaded into and / or executed by) a non-transitory machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine (e.g., a computer or processor), the machine becomes an apparatus for practicing the patented invention(s). When implemented on a general-purpose processor, the program code segments combine with the processor to provide a unique device that operates similarly to specific logic circuits.

[0105] Unless expressly stated otherwise, each numerical value and range should be interpreted as being approximate as if the value or range were preceded by the word "about" or "approximately."

[0106] The use of figure numbers and / or reference numerals in the claims is intended to identify one or more possible embodiments of the claimed subject matter to facilitate interpretation of the claims. Such use should not be construed as necessarily limiting the scope of the claims to the embodiments shown in the corresponding figures.

[0107] Although elements in the method claims (if any) are recited in a specific order with corresponding labels, these elements are not necessarily intended to be implemented in that specific order unless the claim recitation otherwise implies a specific order for implementing some or all of these elements.

[0108] Reference herein to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present disclosure. The various appearances of the phrase "in one embodiment" in this specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. The same applies to the term "implementation."

[0109] Unless otherwise specified herein, the use of ordinal adjectives "first," "second," "third," etc. to refer to an object among a plurality of similar objects indicates merely that different instances of such similar objects are being referred to and is not intended to imply that the similar objects so referred to are necessarily in a corresponding order or sequence, whether temporally, spatially, in ranking, or in any other manner.

[0110] Unless otherwise specified herein, the conjunction "if" may also or alternatively be interpreted to mean, in addition to its plain meaning, "when" or "at the time" or "in response to determining" or "in response to detecting," which interpretation may depend on the specific context. For example, the phrase "if it is determined that" or "if [stated condition] is detected" may be interpreted to mean "upon determining that" or "in response to determining that" or "upon detecting [stated condition or event]" or "in response to detecting [stated condition or event]."

[0111] Also for the purposes of this description, the terms "couple," "coupling," "coupled," "connect," "connecting," or "connected" refer to any means known in the art or later developed that allow energy to be transferred between two or more elements, and the insertion of one or more additional elements is contemplated, although not required. In contrast, the terms "directly coupled," "directly connected," etc., imply the absence of such additional elements.

[0112] As used herein with respect to components and standards, the term "compliant" means that the component communicates with other components in a manner specified in whole or in part by the standard and will be recognized by other components as sufficiently capable of communicating with other components in the manner specified by the standard. A compliant component is not required to operate internally in the manner specified by the standard.

[0113] The functions of the various elements shown in the figures, including any functional blocks labeled "processor" and / or "controller," can be provided using dedicated hardware and hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, a single shared processor, or multiple individual processors (some of which may be shared). Furthermore, the explicit use of the terms "processor" or "controller" should not be interpreted as referring exclusively to hardware capable of executing software and may implicitly include, but not be limited to, digital signal processor (DSP) hardware, network processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), read-only memory (ROM), random access memory (RAM), and non-volatile storage devices for storing software. Other conventional and / or custom hardware may also be included. Similarly, any switches shown in the figures are conceptual only. Their functions may be performed through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, with the specific technique being selectable by the implementer as more specifically understood from the context.

[0114] As used in this application, the terms "circuit" and "circuitry" may refer to one or more or all of the following: (a) a purely hardware circuit implementation (e.g., implementation in purely analog circuitry and / or digital circuitry); (b) a combination of hardware circuitry and software, such as (as applicable): (i) a combination of analog hardware circuitry and / or digital hardware circuitry with software / firmware, and (ii) any portion of a hardware processor with software (including digital signal processor(s), software, and memory(s) that function together to enable a device such as a mobile phone or server to perform various functions); and (c) hardware circuit(s) and / or processor(s), such as microprocessor(s) or portions of microprocessor(s), that require software (e.g., firmware) to operate, but may be absent when not required for operation. This definition of circuitry applies to all uses of this term in this application, including all claims. As a further example, as used in this application, the term "circuitry" also covers implementations that are merely a hardware circuit or a processor (or processors), or a portion of a hardware circuit or processor, and their accompanying software and / or firmware. The term "circuitry" also covers, for example, a baseband integrated circuit or a processor integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or networking device, if applicable to a particular claim element.

[0115] Those skilled in the art will recognize that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the present disclosure. Similarly, it will be recognized that any flow charts, flow diagrams, state transition diagrams, pseudocode, etc. represent various processes that can be substantially represented in a computer-readable medium and thus executed by a computer or processor, whether or not the computer or processor is explicitly shown.

[0116] The Summary of the Invention in this specification is intended to introduce some example embodiments, and additional embodiments are described in the Detailed Description and / or with reference to one or more of the accompanying drawings. The Summary is not intended to identify essential elements or features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

Claims

1. An image processing system for encoding spherical sampling data, the image processing system comprising: A processor, the processor being configured to perform data encoding, the processor being configured to: Receiving a three-dimensional environment; For a set of cubemap texture coordinates, applying a cubemap function to the cubemap texture coordinates to generate cubemap directions; applying a low-discrepancy sequence sampling function to the cubemap directions to generate low-discrepancy spherical sampling directions; applying an offset cubemap function to the low-discrepancy spherical sampling directions to generate offset sampling directions; and Pixel values ​​for the cubemap texture coordinates are generated using the offset sampling direction and the three-dimensional environment.

2. The image processing system according to claim 1, wherein The low-discrepancy sequence sampling function is a uniform cubic sampling function.

3. The image processing system according to claim 1, wherein: The low-discrepancy sequence sampling function is an equicubic uniform sampling function.

4. The image processing system according to any one of claims 1 to 3, wherein: The processor is further configured to provide the pixel values ​​of the cubemap texture coordinates via a virtual reality device.

5. The image processing system according to any one of claims 1 to 4, wherein: The processor is configured to apply the offset cube mapping function by performing operations comprising: receiving a gaze vector indicating a viewing direction of a user; determining a rate of change of the gaze vector; and An offset value is selected based on the rate of change of the gaze vector.

6. The image processing system according to claim 5, wherein: Selecting the offset value includes: When the rate of change of the gaze vector is less than a first threshold, selecting a first offset value; and When the change rate of the gaze vector is greater than or equal to the first threshold, a second offset value is selected, wherein the first offset value is greater than the second offset value.

7. The image processing system according to any one of claims 1 to 4, wherein: The processor is further configured to: receiving a gaze vector indicating a viewing direction of a user; and The offset sampling direction is aligned with the gaze vector.

8. The image processing system according to any one of claims 1 to 7, wherein: The processor is further configured to select a sampling frequency of the low-discrepancy sequence sampling function.

9. The image processing system according to any one of claims 1 to 8, wherein: The processor is configured to apply the cubemap function to the cubemap texture coordinates in a shader.

10. An image processing system for encoding spherical sampling data, the image processing system comprising: A processor, the processor being configured to perform data encoding, the processor being configured to: Receiving a three-dimensional environment; Applying an offset low-discrepancy sequence (OLDS) sampling function to a set of cubemap texture coordinates to generate OLDS directions; and Pixel values ​​of the cubemap texture coordinates are generated using the OLDS direction and the three-dimensional environment.

11. The image processing system according to claim 10, wherein: The OLDS sampling function includes a uniform cube sampling function.

12. The image processing system according to claim 10, wherein: The OLDS sampling function includes an equicubic uniform sampling function.

13. The image processing system according to any one of claims 10 to 12, wherein: The processor is configured to apply the OLDS sampling function by performing operations comprising: receiving a gaze vector indicating a viewing direction of a user; determining a rate of change of the gaze vector; selecting an offset value based on the rate of change of the gaze vector; and An offset cubemap function is applied using the offset value.

14. The image processing system according to claim 13, wherein: Selecting the offset value includes: When the rate of change of the gaze vector is less than a first threshold, selecting a first offset value; and When the change rate of the gaze vector is greater than or equal to the first threshold, a second offset value is selected, wherein the first offset value is greater than the second offset value.

15. The image processing system according to any one of claims 10 to 14, wherein: The processor is further configured to: receiving a gaze vector indicating a viewing direction of a user; and The offset sampling direction of the OLDS sampling function is aligned with the gaze vector.

16. The image processing system according to any one of claims 10 to 15, wherein: The processor is further configured to: Select the sampling frequency of the OLDS sampling function.

17. An image processing system for decoding spherical sampling data, the image processing system comprising: A processor, the processor being configured to perform data decoding, the processor being configured to: receiving an offset sampled cubemap from an encoded bitstream; applying an inverse offset cubemap function to the offset sampling direction to generate a low-discrepancy spherical sampling direction; applying an inverse low-discrepancy sequence sampling function to the low-discrepancy spherical sampling directions to generate cubemap directions; applying an inverse cubemap function to the cubemap directions to generate cubemap texture coordinates; and A pixel value for the offset sampling direction is generated using the cubemap texture coordinates and the offset sampling cubemap.

18. The image processing system according to claim 17, wherein: The inverse low-discrepancy sequence sampling function is an inverse uniform cube sampling function.

19. The image processing system according to claim 17, wherein: The inverse low-discrepancy sequence sampling function is an inverse equicubic sampling function.

20. The image processing system according to any one of claims 17 to 19, wherein: The processor is further configured to: Metadata associated with the offset sample cubemap is received from the encoded bitstream, wherein the metadata includes the inverse offset cubemap function.

Images