Application of Layered Coding in Distributed Computing

Abstract

A network system for generating a sequence of frames for rendering a dynamic 3D scene, the system including a first rendering node and a second rendering node, the first rendering node configured to generate a sequence of first partially rendered data, perform layer encoding on each of the first partially rendered data to generate a sequence of encoded first partially rendered data, and transmit the sequence of encoded first partially rendered data to the second rendering node, the second rendering node configured to obtain the sequence of encoded first partially rendered data from the first rendering node and generate a sequence of second partially or fully rendered frames based on the sequence of encoded first partially rendered data.

Description

Technical Field

[0001] The following disclosure relates to a system in which an image is generated and displayed on a display device. The image can, for example, virtually represent an object in a 3D space. The display device can be, for example, a user extended reality (“XR”, including virtual reality and / or augmented reality) headset, XR smart glasses, an autostereoscopic display, a TV display, a mobile device, a PC, etc.

Background Art

[0002] According to so-called “split computing” or “remote rendering”, an image is generally generated away from the display device, and there are usually limitations on the communication speed and capacity between the image source and the display device. For example, the image source and the display device may be connected via a network, and the image source can be arranged, for example, on another device in the same room, a server in a nearby private data center, or a server in a so-called cloud.

[0003] Due to limitations on communication speed and capacity, as well as the processing speed and processing resources required for image generation, it is desirable to minimize elements such as frame rate and transmission data volume as much as possible without degrading the user experience. Furthermore, the bandwidth may change over time, and it may be desirable to immediately reduce the transmission data volume when the bandwidth decreases in order to avoid so-called “latency jitter”. Although it is common in the art to adapt the resolution and / or frame rate of a video sequence transmitted to a display device, the response time of variable resolution and frame rate usually lags behind a sudden decrease in bandwidth (especially in the case of wireless transmission), and the requirement to send a relatively large I-frame (e.g., an intra-frame independent of the preceding frame) each time the resolution changes further increases the risk of latency jitter due to an unreliable data channel.

[0004] Furthermore, due to the time required for image generation and the transmission of the image to the display device, there may be a difference between the display required at the time of image generation (the "viewport") and the display required at the time of image display on the display device. As a non-limiting example, when the display device is a user headset, the user may move their head either intentionally or unconsciously.

[0005] These problems are particularly difficult when high-quality 3D video (e.g., 3D meshes or point clouds) is expected (e.g., in the near-photorealistic graphics of modern video games), and when the headset is required to be low-weight and low-power (e.g., a target consumption of 1 watt).

[0006] One well-known technique for coping with limited frame rates and the latency between image generation and image display is known as reprojection, warping, or temporal warping.

[0007] Furthermore, it is also known to use depth maps to assist in the representation of 3D space. A depth map indicates how far different surfaces of a 3D object or different parts of an image appear to be from the viewer. Depth maps can be used in time warping (which is often referred to as "spatial warping" since parallax can be considered when such depth assistance is provided), as well as in other real-time display adjustments such as variable focus correction based on gaze tracking. Depth maps are often used to assist spatial warping when rendering is executed locally within the same device and accurate depth information is directly available. However, when rendering is executed according to a split computing method, it is currently difficult to efficiently transmit depth information compressed at a sufficient bit depth granularity. This is particularly due to the fact that there is insufficient bandwidth, computing power is limited, and the hardware acceleration methods available for compressing image and video data provide a maximum pixel data accuracy of 10-bit values and are thus relatively inaccurate for the purpose of depth information.

[0008] By making the video streaming component of split computing more efficient, it is conceivable that the use cases supported by split computing will increase, and thus the need to make the rendering process as efficient as possible is emphasized. According to so-called "hybrid rendering", it has been proposed to further split the rendering process itself across multiple servers and not necessarily locate them in the same place. This can be done, for example, by performing computationally expensive calculations on one server and then completing viewport rendering on another low-power device closer to the user. A known problem in the art is how to suitably execute hybrid rendering and then efficiently compress the intermediate by-products and transmit them to the device that performs the final rendering of the viewport. SUMMARY OF THE INVENTION

[0009] According to a first aspect, the present disclosure provides a network system for generating a sequence of frames for rendering a dynamic 3D scene. The system includes a first rendering node and a second rendering node. The first rendering node generates a sequence of first partially rendered frames, performs encoding (in particular, as a non-limiting example, layer encoding) on each of the first partially rendered frames to generate a sequence of encoded first partially rendered frames, and is configured to transmit the sequence of encoded first partially rendered frames to the second rendering node. The second rendering node is configured to obtain the sequence of encoded first partially rendered frames from the first rendering node and generate a sequence of second partially or fully rendered frames based on the sequence of encoded first partially rendered frames. For example, the first rendering node may be a node of a distributed rendering network, and the second rendering node may be another node of the distributed rendering network or a user device, such as a VR headset. The first rendering node may be a central node, and the second rendering node may be an edge node. The rendering node may serve multiple users, and the first rendering node may serve more users than the second rendering node.

[0010] Optionally, in the case of the first aspect, the second rendering node is configured to decode the sequence of encoded first partially rendered frames to obtain the sequence of first partially rendered frames.

[0011] Optionally, in the case of the first aspect, the second rendering node is configured to perform layer encoding on each second partially or fully rendered frame to generate a sequence of encoded second partially or fully rendered frames.

[0012] In one embodiment, the first rendering node is configured to perform layer encoding according to a first encoding method, the second rendering node is configured to perform layer encoding according to a second encoding method, and the first encoding method is different from the second encoding method.

[0013] Optionally, in the case of the first aspect, the frame rate for the sequence of second partially or fully rendered frames is greater than the frame rate for the sequence of first partially rendered frames. In a non-limiting embodiment, this is achieved by the partially rendered frames containing data to be processed while performing interpolation of the frame rate to support more accurate interpolation.

[0014] Optionally, in the case of the first aspect, the first rendering node generates a sequence of first partially rendered frames containing a first data type, the second rendering node generates a sequence of second partially or fully rendered frames containing a second data type, and the first data type is different from the first data type. For example, the first data type may be point cloud data, the second data type may be image data, and the image data is generated at least partially based on the point cloud data.

[0015] Optionally, in the case of the first aspect, the system includes a display device, and the communication delay between the first rendering node and the display device is greater than the communication delay between the second rendering node and the display device.

[0016] In one embodiment, the first rendering node is configured to obtain a viewing position from a display device before generating a sequence of first partially rendered frames, and the second rendering node is configured to obtain an updated viewing position from the display device before generating a sequence of second partially or fully rendered frames.

[0017] Optionally, in the first aspect, generating a sequence of first partially rendered frames requires more processing resources than generating a sequence of second partially or fully rendered frames.

[0018] Optionally, in the first aspect, each frame includes image data and depth map data, and the system includes a third rendering node configured to generate a sequence of third fully rendered frames by performing temporal warping and / or depth correction on the sequence of second partially or fully rendered frames using the depth map data. For example, the first rendering node and the second rendering node may be nodes of a distributed rendering network, and the third rendering node may be a user device, such as a VR headset. The first rendering node may be a central node, and the second rendering node may be an edge node. The rendering node may serve multiple users, and the first rendering node may serve more users than the second rendering node.

[0019] Optionally, in the first aspect, each frame includes point cloud data in which each of a plurality of points has a 3D position and one or more attributes.

[0020] When each frame includes point cloud data, the second or third rendering node may be configured to calculate depth map data based on the 3D positions of the points.

[0021] Optionally, in the case of the first aspect, the first rendering node is configured to generate one or more sequences of first partially rendered frames for a first number of users or display devices, and the second rendering node is configured to generate one or more sequences of second partially or fully rendered frames for a second number of the plurality of users or display devices, the second number being less than the first number.

[0022] Optionally, in the case of the first aspect, the network system dynamically selects, in response to at least one metric including a metric of the complexity of the rendering tasks to be performed, the available free capacity at each node, the location of the display device relative to the node's network, the round-trip latency between the node and the display device, the available bandwidth between nodes and from nodes to display devices, and the number of separate display devices that require rendering of the same 3D scene within the viewing range, how many nodes and which specific nodes to use in the rendering process.

[0023] Optionally, in the case of the first aspect, the partially rendered frame is encoded as volumetric data including, by way of non-limiting example, point cloud data, mesh data, and / or texture data, enabling subsequent rendering node(s) to render multiple viewpoints within a range (a "zone of view"), and the partially rendered frame is used by one or more subsequent rendering nodes to generate fully rendered frames for at least two users. Thereby, the computationally intensive aspects of the scene can be calculated only once for multiple users.

[0024] Optionally, in the case of the first aspect, the partially rendered frame includes light field data, enabling these environmental characteristics to be calculated only once for multiple users in the same virtual environment.

[0025] Optionally, in the case of the first aspect, the partially rendered frame includes spatial characteristics that enable the calculation of the behavior of sound in 3D space, and it is possible to calculate these characteristics only once for multiple users in the same virtual environment.

[0026] Optionally, in the case of the first aspect, the partially rendered frame is encoded using at least a partially irreversible encoding method.

[0027] Optionally, in the case of the first aspect, the partially rendered frame is encoded using at least a layered encoding method.

[0028] Optionally, in the case of the first aspect, subsequent nodes receive only a portion of the data encoded by the first rendering node in response to a specific location of one or more viewpoints of the fully rendered frame to be calculated.

[0029] Optionally, in the case of the first aspect, subsequent nodes decode only a subset of the encoded data generated by the first rendering node and received by the subsequent nodes that is necessary to fully render the specific field of view that it is rendering at any given point.

[0030] Optionally, in the case of the first aspect, the partially rendered frame is encoded by at least partially using a point cloud format that represents points according to one or more coordinate systems, and each point is assigned one or more data attributes that specify visual characteristics including one or more of size, normal vector, motion information, color information, and transparency.

[0031] According to aspects related to the first aspect, the present disclosure provides a method for generating a sequence of frames for rendering a dynamic 3D scene. A first rendering node of a network system generates a sequence of first partially rendered frames, performs encoding (in particular, as a non-limiting example, layered encoding) on each of the first partially rendered frames to generate a sequence of encoded first partially rendered frames, and transmits the sequence of encoded first partially rendered frames to a second rendering node of the network system. The second rendering node acquires the sequence of encoded first partially rendered frames from the first rendering node and generates a sequence of second partially or fully rendered frames based on the sequence of encoded first partially rendered frames. Optional features of the first aspect may be applied to related aspects.

[0032] According to aspects related to the first aspect, the present disclosure provides a bitstream including a sequence of encoded first partially rendered frames. The bitstream is generated by a first rendering node generating a sequence of first partially rendered frames and performing encoding (in particular, as a non-limiting example, layered encoding) on each of the first partially rendered frames to generate a sequence of encoded first partially rendered frames. The bitstream is suitable for a second rendering node to generate a sequence of second partially or fully rendered frames based on the sequence of encoded first partially rendered frames. Optional features of the first aspect may be applied to related aspects.

[0033] According to a second aspect, the present disclosure provides a method for encoding a sequence of frames representing a dynamic 3D scene, each frame including base layer image data and enhancement data, the method including performing layer encoding on a frame of the sequence of frames to generate an encoded frame including a base image layer and an enhancement image layer.

[0034] Optionally, in the case of the second aspect, the enhancement image layer includes data used by a decoder device to reconstruct a higher resolution representation of the sequence of frames.

[0035] Optionally, in the case of the second aspect, the enhancement image layer includes data used by a decoder device to reconstruct a higher bit depth representation of the sequence of frames relative to the bit depth of the base image layer.

[0036] Optionally, in the case of the second aspect, the enhancement image layer includes data used by a decoder device to reconstruct the distance of objects within the image from the viewer.

[0037] Optionally, in the case of the second aspect, the enhancement image layer includes data used by a decoder device to reconstruct tactile feedback for the user.

[0038] Optionally, in the case of the second aspect, the method further includes discarding the enhancement data during transmission of the sequence of frames in response to a reduction in the transmission channel bandwidth, indicating to the encoder that the enhancement data has been dropped, refreshing the temporal buffer for the enhancement data encoding if the enhancement data has been dropped, and performing an instantaneous decoder refresh (IDR) for the enhancement data (but not necessarily for the base layer data) to account for the decoder having missed some of the previous enhancement data.

[0039] Optionally, in the case of the second aspect, the layer coding method used is MPEG-5's LCEVC (Low Complexity Enhanced Video Coding) coding or SMPTE's VC-6 coding.

[0040] Optionally, in the case of the second aspect, at least one of the enhancement data is transmitted as embedded user data within the coefficients of the LCEVC data. As a non-limiting example, one or more residual coefficients of an image include embedded depth information representing the depth of the corresponding object with respect to the viewpoint, and the decoder processes the embedded data to reconstruct a depth map associated with the image frame based at least in part on the embedded data. In a non-limiting embodiment, the depth map is reconstructed by processing both the embedded data and the image data.

[0041] Optionally, in the case of the second aspect, or as a stand-alone implementation, the frames plus depth data sent at a given frame rate are used by the display device to increase the frame rate via spatial warping (depth-based reprojection) to match the display frame rate, so that the rendering and video streaming processes can be run at a frame rate lower than the display frame rate and the required bandwidth and processing resources can be reduced.

[0042] Optionally, in the case of the second aspect, each frame includes image data and depth map data, and the method includes performing layer coding on the frames of the sequence of frames to generate an encoded frame including one or more of a base depth map layer and an enhancement depth map layer.

[0043] Optionally, in the case of the second aspect, each frame includes an image layer and a depth map layer, and the encoded frame includes a base image layer, a base depth map layer, an enhancement image layer, and an enhancement depth map layer.

[0044] Optionally, in the case of the second aspect, each frame includes depth map data embedded within the image data, the base depth map layer is a base image layer having the embedded depth map data, and the enhanced depth map layer is an enhanced image layer having the embedded depth map data.

[0045] Optionally, in the case of the second aspect, the method further includes receiving a depth map drop indication indicating whether depth map data has been dropped when transmitting a sequence of frames, and, if the depth map data has been dropped and the available bandwidth is lower than a threshold, discarding the depth map data of the frame and performing layered encoding on the image data of the frame to generate an encoded frame including a base image layer and an enhanced image layer.

[0046] According to an aspect related to the second aspect, the present disclosure provides an encoder configured to encode a sequence of frames representing a dynamic 3D scene, each frame including base layer image data and enhancement data, and the encoder is specifically configured to perform layered encoding on the frames of the sequence of frames to generate an encoded frame including a base image layer and an enhanced image layer. Optional features of the second aspect may be applied to related aspects.

[0047] According to an aspect related to the second aspect, the present disclosure provides a bitstream including an encoded sequence of frames representing a dynamic 3D scene, each frame including a base image layer and an enhanced image layer.

[0048] According to a third aspect, the present disclosure provides a method for decoding a sequence of frames representing a dynamic 3D scene, each frame including base layer image data and enhancement data, the method including performing layered decoding on a frame of the sequence of frames to generate a decoded frame from the base image layer and the enhanced image layer.

[0049] Optionally, in the case of the third aspect, the enhanced image layer contains data used in the decoding method to reconstruct a higher resolution representation of the sequence of frames.

[0050] Optionally, in the case of the third aspect, the enhanced image layer contains data used in the decoding method to reconstruct a higher bit-depth representation of the sequence of frames relative to the bit-depth of the base image layer.

[0051] Optionally, in the case of the third aspect, the enhanced image layer contains data used in the decoding method to reconstruct the distance of objects within the image from the viewer.

[0052] Optionally, in the case of the third aspect, the enhanced image layer contains data used in the decoding method to reconstruct tactile feedback for the user.

[0053] Optionally, in the case of the third aspect, the method further includes sending an indication that enhanced data should be dropped.

[0054] Optionally, in the case of the third aspect, the layer decoding method used is MPEG-5's LCEVC (Low Complexity Enhanced Video Coding) coding or SMPTE's VC-6 coding.

[0055] Optionally, in the case of the third aspect, at least one of the enhanced data is received as embedded user data within the coefficients of the LCEVC data. As a non-limiting example, one or more residual coefficients of the image contain embedded depth information representing the depth of the corresponding object relative to the viewpoint, and the decoder processes the embedded data to reconstruct a depth map associated with the image frame based at least in part on the embedded data. In non-limiting embodiments, the depth map is reconstructed by processing both the embedded data and the image data.

[0056] Optionally, in the case of the third aspect, or as a stand-alone implementation, the frames plus depth data sent at a given frame rate are used by the display device to increase the frame rate via spatial warping (depth-based reprojection) to match the display frame rate, such that the rendering and video streaming processes can be run at a frame rate lower than the display frame rate and the required bandwidth and processing resources can be reduced.

[0057] Optionally, in the case of the third aspect, each frame includes image data and depth map data, and the method includes performing layer decoding on a frame of a sequence of frames to produce an encoded frame including one or more of a base depth map layer and an enhanced depth map layer.

[0058] Optionally, in the case of the third aspect, each frame includes an image layer and a depth map layer, and the encoded frame includes a base image layer, a base depth map layer, an enhanced image layer, and an enhanced depth map layer.

[0059] Optionally, in the case of the third aspect, each frame includes depth map data embedded within the image data, the base depth map layer is the base image layer with the embedded depth map data, and the enhanced depth map layer is the enhanced image layer with the embedded depth map data.

[0060] Optionally, in the case of the third aspect, the method further includes transmitting a depth map drop indication indicating that depth map data should be dropped when sending a sequence of frames, and when depth map data has been dropped, performing layer decoding on the image data of the frames to produce a decoded frame from the base image layer and the enhanced image layer.

[0061] In an aspect related to the third aspect, a decoder or a display device configured to execute the method according to the third aspect is provided by the following disclosure. Optional features of the third aspect may be applied to the related aspect.

[0062] According to a fourth aspect, a method for encoding a sequence of frames representing a dynamic 3D scene is provided by the present disclosure, each frame including image data and additional data, the method including performing layered encoding on a frame of the sequence of frames to generate an encoded frame including a base layer and an enhancement layer. In particular, according to an embodiment of the fourth aspect, a method for encoding a sequence of frames representing a dynamic 3D scene is provided by the present disclosure, each frame including image data and depth map data, the method including performing layered encoding on a frame of the sequence of frames to generate an encoded frame including a base depth map layer and an enhancement depth map layer.

[0063] In an aspect related to the fourth aspect, an encoder or a renderer configured to execute the method according to the fourth aspect is provided by the following disclosure. Optional features of the fourth aspect may be applied to the related aspect.

[0064] According to a fifth aspect, a method for decoding a sequence of frames representing a dynamic 3D scene is provided by the present disclosure, each frame including image data and depth map data, the method including identifying one or more object regions in the image from the image data and assigning depth information to each of the one or more object regions in the image based on the depth map data. In some embodiments, the depth map data includes a base depth map layer and an enhancement depth map layer, and decoding each frame includes performing layered decoding using the image data, the base depth map layer, and the enhancement depth map layer.

[0065] In aspects related to the fifth aspect, a decoder or display device configured to execute the method according to the fifth aspect is provided by the following disclosure. Optional features of the fifth aspect may be applied to related aspects.

[0066] According to a sixth aspect, by the following disclosure, there is provided a method including receiving frame plus depth data at a given frame rate encoded by a layered encoding scheme, and increasing the frame rate in a display device via depth-based reprojection using the depth data to match a display frame rate.

[0067] Optionally, in the case of the sixth aspect, the frame includes data representing a dynamic 3D scene, and each frame includes base layer image data and enhancement data. Optionally, in the case of the sixth aspect, the method includes performing layered decoding on a frame of a sequence of frames to generate a decoded frame from an encoded frame including a base image layer and an enhancement image layer.

[0068] In aspects related to the sixth aspect, a decoder or display device configured to execute the method according to the sixth aspect is provided by the following disclosure. Optional features of the sixth aspect may be applied to related aspects.

[0069] According to a seventh aspect, by this disclosure, a bit sequence representing the encoding of a sequence of frames representing a dynamic 3D scene is provided, the bit sequence including encoded data for a base depth map layer for a frame and encoded data for an enhancement depth map layer for the frame.

[0070] Optionally, in the case of the seventh aspect, the bit sequence further includes encoded data for a base image layer for a frame and encoded data for an enhancement image layer for the frame.

[0071] Optionally, in the case of the seventh aspect, the base depth map layer is a base image layer having embedded depth map data, and the enhanced depth map layer is an enhanced image layer having embedded depth map data.

[0072] According to an eighth aspect, there is provided, by the present disclosure, a method for transmitting a sequence of frames representing a dynamic 3D scene, each frame including image data and depth map data, the method including obtaining an encoded frame including a base depth map layer and an enhanced depth map layer, determining whether to drop the depth map data when transmitting the sequence of frames, and if the depth map data is to be dropped, discarding the depth map data from the encoded frame to give a reduced encoded frame including a base image layer and an enhanced image layer, and transmitting the reduced encoded frame, and if not, transmitting the encoded frame.

[0073] Optionally, in the case of the eighth aspect, each frame includes an image layer and a depth map layer, the encoded frame includes a base image layer, a base depth map layer, an enhanced image layer, and an enhanced depth map layer, and discarding the depth map data includes discarding the enhanced depth map layer, and preferably further includes discarding the base depth map layer.

[0074] Optionally, in the case of the eighth aspect, each frame includes depth map data embedded within the image data, the base depth map layer is a base image layer having embedded depth map data, the enhanced depth map layer is an enhanced image layer having embedded depth map data, and discarding the depth map data includes removing the embedded depth map data from the enhanced image layer, and preferably further includes removing the embedded depth map data from the base image layer.

[0075] Optionally, in the case of the eighth aspect, the method further includes generating a depth map drop indication indicating whether depth map data is dropped when transmitting a sequence of frames, and sending the depth map drop indication to an upstream encoder or sending the depth map drop indication together with an encoded frame or a downscaled encoded frame.

[0076] In an aspect related to the eighth aspect, a transmitter configured to execute the method according to the eighth aspect is provided by the following disclosure. Optional features of the eighth aspect may be applied to related aspects.

[0077] According to a ninth aspect, in the following disclosure, a method executed by an encoder for encoding a sequence of frames is provided. The method includes performing layered encoding on a first frame of the sequence of frames to generate a first encoded frame including a base layer and an enhancement layer, storing components of the enhancement layer of the first encoded frame in a temporal buffer for use in subsequent temporal encoding of frames, sending the first encoded frame to a transmitter for transmission, receiving an enhancement drop indication indicating whether the enhancement layer was dropped when transmitting the first encoded frame, and performing layered encoding on a second frame of the sequence of frames to generate a second encoded frame including a base layer and an enhancement layer. When the enhancement drop indication indicates that the enhancement layer was not dropped, the enhancement layer of the second encoded frame is generated with reference to the temporal buffer. When the enhancement drop indication indicates that the enhancement layer was dropped, the enhancement layer of the second encoded frame is generated without reference to the temporal buffer.

[0078] Optionally, in the case of the ninth aspect, the method further includes clearing the temporal buffer when the enhancement drop indication indicates that the enhancement layer was dropped.

[0079] Optionally, in the ninth aspect, if no enhancement drop indication is received within a predetermined time limit, the method further includes performing layered encoding on a second frame of the frame sequence to generate a second encoded frame including a base layer and an enhancement layer, and the enhancement layer of the second encoded frame is generated by referring to a time buffer.

[0080] Optionally, in the ninth aspect, if no enhancement drop indication is received within a predetermined time limit, the method further includes performing layered encoding on a second frame of the frame sequence to generate a second encoded frame including a base layer and an enhancement layer, and the enhancement layer of the second encoded frame is generated without referring to a time buffer.

[0081] Optionally, in the ninth aspect, generating an enhancement layer of a frame without referring to a time buffer includes decoding a base layer of the encoded frame and calculating a residual as a difference between the decoded base layer and the frame. <G

[0082] Optionally, in the ninth aspect, generating an enhancement layer of a frame by referring to a time buffer includes decoding a base layer of the encoded frame, calculating a residual as a difference between the decoded base layer and the frame, and calculating a difference between the residual of the current frame and the corresponding residual of the previous frame stored in the time buffer.

[0083] Optionally, in the ninth aspect, the layered encoding is LCEVC encoding.

[0084] In an aspect related to the ninth aspect, the following disclosure provides an encoder configured to execute the method according to the ninth aspect. Optional features of the ninth aspect may be applied to related aspects.

[0085] Although various aspects (in particular the first aspect) have been described in relation to one or more "frames", it should be understood that the data placed within a "frame" is optional, and thus the data may be placed in other formats and still be considered as forming part of the present disclosure.

[0086] Different variations of different aspects may be combined in any combination. An aspect may be omitted and / or modified by features. Particular aspects not described above may be provided by combining any of the individual features described in relation to any of the above and / or below aspects, variations, or implementations.

[0087] Any of the methods described above may be executed by one or more processors that execute a computer program. The computer program includes computer-readable instructions that, when executed by a processor, cause the processor to execute the corresponding method. The computer-readable instructions may be stored within a non-transitory computer-readable medium. The computer-readable instructions may be encoded in a digital signal (e.g., an optical signal or an electrical signal).

[0088] Furthermore, for any of the methods described above that generate or consume an encoded sequence of frames, the encoded sequence of frames may be separated as a bitstream, and this bitstream can be transmitted to a final destination for the user to consume immediately after rendering or after encoding, or can be stored in memory for an indefinite period. For example, the bitstream may be stored by a streaming service as part of a video-on-demand service.

Brief Description of the Drawings

[0089]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

DETAILED DESCRIPTION OF THE INVENTION

[0090] By non-limiting, exemplary embodiments disclosed herein, a method is provided for structuring rendering resources in a hierarchy of possible available resources (defined herein as a “Content Rendering Network” or “CRN” by analogy to a content delivery network that provides a hierarchical caching infrastructure for web content). By way of non-limiting example, based on real-time metrics including the expected processing load of the rendering to be performed, the state of the available resources within the CRN, the network latency from resources within the CRN, and the number of other users requesting rendering of the same 3D space from within the scope of view, the CRN can dynamically select one or more resources for performing the rendering process, and, if it is optimal to do so, can efficiently pool the costly rendering calculations to be performed once for multiple users, and then distribute the compressed by-products to another resource (either another node of the CRN or a user device) that performs the rendering of the final viewport and streaming to the display device.

[0091] The non-limiting additional exemplary embodiments disclosed herein provide a method for optimizing the transmission of ultra-low latency video frames by a layered coding scheme (i.e., a method including a layer of data representing a signal at a lower quality level and one or more additional layers of data including information for reconstructing the representation of the signal at a higher quality level). As a non-limiting example, the additional layer of data enables a decoder device to enhance one or more quality aspects of the signal, including visual fidelity, resolution, bit depth of the image data (e.g., for high-nit HDR), frame rate, stereoscopy, focal distance (e.g., for variable focus adjustment), haptics, etc. One or more of the additional data layers can be treated as optional and can be safely dropped after encoding even in the middle of its transmission without degrading the quality of the base layer data and without requiring the transmission of costly base layer I frames (since usually only a much smaller enhanced data IDR is required). Thus, layered coding not only enables the transmission of enhanced video within the same bandwidth, but importantly provides additional degrees of freedom in how to degrade quality when bandwidth is insufficient, as well as providing a more rapid dynamic adaptation to bandwidth degradation.

[0092] Figures 1 and 2 schematically illustrate the use of depth maps and warping when generating and displaying a dynamic 3D scene.

[0093] In Figures 1 and 2, a plurality of image portions 1a, 1b, 1c are shown. These can be displayed simultaneously as part of the overall image.

[0094] Alternatively, they can be displayed sequentially as separate frames. In other words, image portion 1a can correspond to frame N, image portion 1b can correspond to frame N+1, etc.

[0095] In FIG. 1, all of the image portions are mapped to the same plane 3 when being displayed. In other words, in the 3D virtual environment that a user views through a 3D display (e.g., a VR, XR, or AR display), all of the image portions 1a, 1b, 1c are displayed as if they were at the same distance from the viewer.

[0096] On the other hand, in FIG. 2, each image portion 1a, 1b, 1c is mapped to a respective virtual plane 3a, 3b, 3c (which may be at different distances from the viewer) within the 3D virtual environment.

[0097] In a 3D environment, a plane at a certain distance from the viewer is curved, and planes at different distances have different curvatures. As a result, various visual effects, such as time warping and focus distance adjustment (which may be local or variable focus and can be used according to eye tracking), behave differently according to the distance between the viewer and the image portion. Thus, by incorporating information regarding the distance between the image portion and the viewer, the 3D environment can be rendered more realistically.

[0098] Such information regarding the distance from the viewer can be incorporated into the frame using a depth map. In the case of a planar image portion, the depth can be specified for each pixel or block of the image portion. Alternatively, if the frame contains point cloud information, or other information regarding the location of points or objects within the 3D virtual environment, this information can be used as equivalent to a depth map when performing the calculation of visual effects.

[0099] Figures 1 and 2 further illustrate the characteristics of time warping. As shown in Figures 1 and 2, each of the display images 1a, 1b, 1c is a part of the area of the warpable images 2a, 2b, 2c. Due to the delay between the rendering and display of the frame, the warpable images 2a, 2b, 2c are rendered before the latest viewing direction of the user is known. The warpable images 2a, 2b, 2c can be sent to the display device or a rendering node close to the display device, and the display device or the rendering node can perform time warping to generate the display image portions 1a, 1b, 1c based on the corresponding warpable images 2a, 2b, 2c and the latest viewing direction of the user.

[0100] Figure 3 schematically illustrates a system for generating an image frame sequence.

[0101] Referring to Figure 3, the system includes an image generator 31, an encoder 32, a transmitter 33, a network 34, a receiver 35, a decoder 36, and a display device 37.

[0102] The image generator 31, the encoder 32, and the transmitter 33 together form a first rendering node.

[0103] The image generator 31 may include, for example, a rendering engine for initially rendering a virtual environment (such as a game or a virtual conference room).

[0104] The image generator 31 is configured to generate a sequence of frames to be displayed. The frames may include, as described above, one or more image portions 1a, 1b, 1c. In addition or alternatively, the frames may include point cloud data. In a point cloud, each point typically has a 3D position and one or more attributes, and the attributes may include, for example, the color of the surface, the transparency value, the size of the object, and the normal direction of the surface. Each attribute may have a value selected from a continuous range or a value selected from a discrete set.

[0105] Each frame may further include depth map data as described above. The depth map data may be provided as a depth map layer separately from the image layer. In some contexts such as MPEG Immersive Video (MIV), the image layer may instead be described as a texture layer. Similarly, in some contexts, the depth map layer may instead be described as a geometry layer.

[0106] Furthermore, each frame may include the location of the predicted display window. The location of the predicted display window is the location of the portion of the generated images 2a, 2b, 2c that may be displayed by the display device 37. The location of the predicted display window may be based on the user's viewing position obtained from the display device 37 (e.g., the virtual position and / or orientation of the user in a 3D environment). The location of the predicted display window may be defined using one or more coordinates. For example, referring to FIG. 1, the location of the predicted display window may be defined using the coordinates of the corner or center of the predicted display window, or may be defined using the size of the predicted display window. The location of the predicted display window may be encoded as part of the metadata included in the frame.

[0107] Each frame may contain additional information, which may be provided as separate layers. For example, each frame may further include audio or tactile feedback information indicating audio or tactile sensations that can be associated with the displayed visual data. An audio layer or a tactile layer may be associated with each frame, and may be omitted for frames where the associated audio or tactile is not required.

[0108] The frame can be based on the state of the virtual environment, the user's position, or the user's viewing direction. Here, the position and viewing direction may be the physical characteristics of the user in the real world, or the position and viewing direction may be purely virtual (for example, controlled using a handheld controller). The image generator 31 can obtain information indicating the user's position, viewing direction, or movement from, for example, the display device 37. In other cases, the generated image may be independent of the user's position and viewing direction. This type of image generation typically requires significant computer resources (e.g., a powerful GPU) and can be implemented within a cloud service or on a local but powerful computer. For example, in the case of a cloud service (e.g., a cloud rendering service (CRN)), the cost per user can be reduced, and as a result, a wide range of users can access image frame generation. Here, "rendering" refers to at least the initial stage of rendering for generating an image. Further rendering can be performed on the display device 37 based on the generated image to generate the final image to be displayed.

[0109] The encoder 32 is configured to encode the frame transmitted to the display device 37. The encoder 32 may be implemented using executable software or may be implemented on specific hardware such as an ASIC.

[0110] The encoder can apply inter-frame or intra-frame compression based on the currently encoded frame and optionally one or more previously encoded frames.

[0111] The encoder 32 can be a multi-layer encoder (e.g., an LCEVC encoder as shown in FIG. 4A or a VC-6 encoder).

[0112] For example, if the generated frame includes depth map data, the coder may perform layered coding on each frame to generate a coded frame including a base depth map layer and an enhancement depth map layer. Similar to the coding of image data, coding depth maps in such a way can improve compression. In some applications such as HDR video, depth maps desirably have a bit depth of up to 12 or 14 bits and are very detailed, which results in a significant increase in the data to be transmitted. As a result, by providing a way to improve the compression of depth maps, it is possible to perform a more realistic depth map-based display when rendering or transmitting the rendered data in real time. Further, this type of layered coding simplifies dropping (and picking up again) one or more of the layers, providing flexibility and tools for bandwidth management.

[0113] Layered coding is also useful because the final decoder / user device (e.g., the user display device) can choose whether to process these extra layers. For example, in a non-layered approach, the best the end device (i.e., the receiver, decoder, or display device associated with the user viewing the frame) can do is determine that it does not have sufficient resources for a given quality (resolution, frame rate, inclusion of depth maps) and signal to the controller / renderer / coder that it does not have sufficient resources. The controller then sends subsequent frames at a lower quality. In the alternative scenario, if the end device can process the received frames, unfortunately, it still has to process the higher quality data until the lower quality data arrives.

[0114] In some of the described embodiments, this situation is improved. Because if the end device determines that it does not have the processing capacity to handle, for example, the highest level of quality, the end device can drop a certain specific layer and / or choose not to process it. Also, the end device can signal to the controller that it requires a lower level of quality, while during that time, the end device can only process as many layers as it can handle. Therefore, the end device can respond much more quickly to the conditions.

[0115] In some cases, depth map data can be embedded within the image data. In this case, the base depth map layer can be the base image layer having the embedded depth map data, and the enhanced depth map layer can be the enhanced image layer having the embedded depth map data.

[0116] Alternatively, if the generated frame includes a depth map layer separate from the image layer and multi-layer encoding is applied, the encoded depth map layer can be separate from the encoded image layer. This has the advantage that the encoded depth map layer can be dropped under some conditions while still retaining the displayable image layer (although the level of realism is reduced). For example, the encoded depth map layer can be dropped by the transmitter or encoder when the available communication resources decrease, or can be dropped by an end device lacking the processing resources to handle the highest level of quality.

[0117] Similarly, if some frames include an audio-based layer, a tactile feedback-based layer, an audio enhancement layer, or a tactile feedback enhancement layer, these can be processed or dropped flexibly.

[0118] In addition or alternatively, if the frame includes point cloud data, the coder may apply point cloud data coding techniques as described in European Patent Application No. EP21386059.6, which is hereby incorporated by reference. Such a point cloud coder may function as a base coder for layered coding techniques such as LCEVC or VC-6. In particular, LCEVC and VC-6 techniques code and decode layered signals, regardless of the content type of the coded data within the signal. For example, the signal may include texture, video frames, geometric or depth data, meshes, point clouds, rendering attributes, or physical engine attributes.

[0119] The transmitter 33 can be any known type of transmitter for wired or wireless communication (e.g., an Ethernet transmitter or a Bluetooth transmitter).

[0120] The transmitter 33 may be configured to determine how to transmit the frame and / or provide feedback to the coder 32 or the image generator 31. For example, the transmitter 33 may determine the communication resources (e.g., bandwidth) available for transmitting the frame, and if the available bandwidth is insufficient to transmit all of the generated data, it may drop one or more layers from the coded frame, or it may instruct the image generator 31 and / or the coder 32 that the frame needs to be generated and coded with fewer layers. As a specific example, the transmitter 33 may be configured to drop the depth map layer, the LCEVC enhancement layer, or the VC-6 enhancement layer from the frame if the available communication resources are insufficient.

[0121] Network 34 is used for communication between the transmitter 33 and the receiver 35 and can be any known type of network, such as a WAN or LAN, or a wireless Wi-Fi or Bluetooth network. Network 34 can further be a composite of several different types of networks. Many users can only access a network with a bandwidth of 30 MBps, so latency jitter may occur during streaming. The required bandwidth and the observed latency can be reduced by tactics such as forward rendering and last millisecond reprojection. These are made possible by improved compression.

[0122] The receiver 35 can be any known type of receiver for wired or wireless communication (e.g., an Ethernet transmitter or a Bluetooth transmitter).

[0123] The decoder 36 is configured to receive and decode the encoded frames. The decoder 36 may be implemented using executable software or on specific hardware such as an ASIC.

[0124] The display device 37 can be, for example, a TV screen or a VR headset. The display timing may be synchronized with the configured frame rate so that the display device 37 can wait before displaying the image. The display device 37 may be configured to perform warping, that is, to obtain the location of the final display window, adjust the warpable images 2a, 2b, 2c, obtain the final images 1a, 1b, 1c corresponding to the user's final viewing direction, and display the final images.

[0125] As described above, the first rendering node may include the image generator 31, the encoder 32, and the transmitter 33. Additional similar rendering nodes may be included in the system and may operate together to generate a sequence of frames.

[0126] In one case, a plurality of rendering nodes may each provide a portion of a sequence of frames to a frame assembly node.

[0127] For example, receiver 35, decoder 36, or display device 37 may be configured to assemble portions of a sequence of frames from multiple sources to generate a final sequence of frames for display.

[0128] Alternatively, the frame assembly node may be separate from receiver 35, decoder 36, and display device 37.

[0129] In addition or alternatively, a plurality of rendering nodes may be chained. In other words, successive rendering nodes may add components as a partial sequence of frames passes from rendering node to rendering node, and ultimately a complete sequence of frames is provided to receiver 35. Further, each rendering node may obtain rendering components from a plurality of upstream rendering nodes and / or distribute rendering components to a plurality of downstream rendering nodes.

[0130] This example is shown in FIG. 3. The second rendering node includes transceiver 33b, decoder 36b, image generator 31b, and encoder 32b. The second rendering node is configured to obtain a first partially rendered frame from the first rendering node and generate a second partially or fully rendered frame based on the first partially rendered frame.

[0131] The image generator 31b may be the same as the image generator 31. Alternatively, the image generator 31b may also be referred to as an image processing device. This is because it processes the first partially rendered frame in order to generate a second partially or fully rendered frame. As described above, the frame may include one or more image portions 1a, 1b, 1c. In addition or alternatively, the frame may include point cloud data.

[0132] The data type of the second partially or fully rendered frame may be different from the data type of the first partially rendered frame. For example, the first partially rendered frame may include point cloud data, and the second partially or fully rendered frame may include image data generated from the point cloud data.

[0133] Alternatively, the first partially rendered frame may include first image data, and the second partially or fully rendered frame may include second image data generated from the first image data.

[0134] Alternatively, the first partially rendered frame may include first point cloud data, and the second partially or fully rendered frame may include second point cloud data generated from the first point cloud data.

[0135] The frame generated by each rendering node may include multiple data types, such as a mixture of image data and point cloud data.

[0136] The second rendering node may operate on the first partially rendered frame in an encoded form (e.g., simply merge the first bitstream generated by the first rendering node with the second bitstream generated by the second rendering node. Alternatively, decoder 36b may decode the first bitstream so that image generator 31b can perform further rendering by referring to the first partially rendered frame. If the further rendering performed by the second rendering node relates only to a portion of the 2D area or 3D volume rendered by the first rendering node, or if the further rendering performed by the second rendering node relates only to the attributes of a portion of the frame rendered by the first rendering node (e.g., if the further rendering does not require color information), decoder 36b may decode only a portion of the data encoded in the first bitstream.

[0137] Similarly, the first rendering node may not be at the head of the chain of rendering nodes. The first rendering node itself may obtain a previously generated partially rendered frame from one or more upstream rendering nodes and generate the first partially rendered frame based on the previously generated partially rendered frame.

[0138] A chain of rendering nodes can be useful for performing different rendering tasks that require different amounts of processing resources or different frame rates. For example, an enterprise can provide distributed processing in the form of a centralized hub with abundant processing resources but far from users and peripheral sites with less processing resources but closer to users. Expensive but fairly static rendering features, such as environmental effects on background lighting or sound, can be generated at a central hub (e.g., using ray tracing), while features that require fewer resources but a faster response or a higher frame rate can be generated closer to the user. In other words, the more the responsiveness of a rendering feature needs to be enhanced, the lower the latency between the rendering node that generates the feature and the user display needs to be, and in the chain of rendering nodes, the node that generates each rendering feature can be selected based on the maximum latency required for that feature. On the other hand, when it is costly to generate a rendering feature, it may be preferable to generate the feature less frequently and with a higher maximum latency. For example, static and high-quality background features may be generated early in the chain of rendering nodes, and dynamic but potentially lower-quality foreground features may be generated later in the chain of rendering nodes, closer to the user device. Here, the environmental effect on sound means that, for example, a series of surfaces can be configured such that each surface has different sound reflection and absorption characteristics depending on the material and shape. The frame rate can be matched by generating multiple frames with features generated at the lower frame rate and combining them with frames with features generated at the higher frame rate. In a non-limiting embodiment, pre-rendering generates volumetric object data including motion vectors at a first (lowest) frame rate, then generates a 2D-rendered frame plus depth information for a specific user at a second (higher) frame rate, and then transmits the video plus depth data to the user device.The user device generates the final frame for display via spatial warping (depth-based reprojection) at the third (highest) frame rate. One or more of these steps may be performed in combination with the other described embodiments. When additional rendering tasks are performed at different rendering nodes in the chain, the user's viewing position may change. Each rendering node or any rendering node may obtain the updated viewing position before performing its respective rendering task.

[0139] Furthermore, the system may simultaneously generate multiple frame sequences for different respective users or different respective display devices. For example, in the context of a VR or AR experience, each user or display device may view different 3D environments or different parts of the same 3D environment. When using a chain of rendering nodes, each node may serve multiple users or just one user.

[0140] For example, the starting rendering node (e.g., in a centralized hub) may serve a large group of users. For example, a group of users may be viewing nearby parts of the same 3D environment. In this case, the starting node may render a wide zone of view (the "field of view") relevant to all users within the large group.

[0141] The starting node may send this broad view to a first intermediate rendering node that renders additional aspects of the 3D environment. These additional aspects may be, for example, aspects that require less processing power for rendering or aspects that are specific to individual users of the group. Further, the intermediate rendering node may render features in a smaller view than the starting node. This smaller view may be related to each user rather than a group of users. The first intermediate rendering node may further serve only a smaller number of users (e.g., half of a large user group), and the remaining users may be served by a second intermediate rendering node that also receives the broad view from the starting node.

[0142] Subsequently, the intermediate rendering node(s) may send a sequence of second partially or fully rendered frames to the end devices for each user. The end devices may optionally use depth map data to perform further processing such as warping or focal length adjustment.

[0143] Preferably, each rendering node encodes the partially or fully rendered frames before sending them to the next rendering node or receiver 35. This means that the required communication resources can be reduced if the rendering nodes are separated by one or more networks or, more generally, if implemented in a distributed system such as the cloud.

[0144] However, each rendering node within the chain encodes the different partially or fully rendered frames with different data. Therefore, it may be advantageous for different rendering nodes to use different rendering formats and / or encoding formats. For example, the output from a first rendering node can be point cloud data that logically describes a 3D scene. This point cloud data can be encoded using the techniques of European patent application No. EP21386059.6. A second rendering node can then operate on the point cloud data to generate image data that can be more easily displayed by a general-purpose display device without the display device having to model the 3D environment. This image data can be encoded using video coding techniques.

[0145] The chain of rendering nodes can be any tree structure that can be extended to any tree structure where a rendering node obtains a partially rendered frame from two or more preceding rendering nodes and generates a further partially or fully rendered frame based on the plurality of obtained sequences of the partially rendered frames.

[0146] For example, a volumetric event may be provided to a number of concurrent users, such as users participating in a shared virtual environment, using a content rendering network (CRN) that includes a number of rendering nodes. Rendering the same event for each user is far more costly in terms of computation time and power consumption than performing a rendering equivalent to rendering the volumetric effect once and multicasting the volumetric effect to multiple users. For example, each user may have a second rendering node (e.g., a VR headset), and the network may include a central first rendering node. The first rendering node may render a volumetric event and distribute a partially rendered frame depicting the volumetric event to different second rendering nodes. The second rendering node for each user may then integrate the partially rendered frame depicting the volumetric event into the view of the virtual environment currently being displayed to each user based on parameters such as the user's virtual position.

[0147] The receiver 35, decoder 36, and display device 37 may be integrated into a single device or separated into two or more devices. For example, some VR headset systems include a base unit and a headset unit that communicate with each other. The receiver 35 and decoder 36 may be incorporated into such a base unit.

[0148] In some embodiments, the network 34 may be omitted. For example, a home display system may include a base unit configured as an image source and a portable display unit including the display device 37.

[0149] If the decoder 36 or the display device 37 does not process or is unable to process one or more layers, the receiver 35 or another transmitter associated with the decoder 36 or the display device 37 may send back the corresponding layer drop indication through the network 34. The layer drop indication may be received by each rendering node. A rendering node that generates a partially or fully rendered frame for that particular decoder 36 or display device 37 may abort the generation of the dropped layer. On the other hand, a rendering node that generates a partially or fully rendered frame for multiple end devices may ignore a layer drop indication received from one end device (since the dropped layer may still be needed for other devices). Alternatively, a rendering node that provides services to multiple end devices may record the received layer drop indication and may abort the generation of the dropped layer only if all the end devices served by the rendering node are to drop the layer.

[0150] In a preferred example, the coder or decoder is part of a hierarchical coding method or format based on a hierarchy. Hierarchical coding enables the transmission of frames at a higher resolution and / or a higher frame rate than what is possible with a single-layer coding method. In hierarchical coding, one or more enhancement layers are transmitted in relation to base data, and the enhancement layers can be used to upsample the base data at the decoder, providing, for example, upsampling in the spatial or temporal dimension. When combined with the equivalent downsampling of the original frame and the generation of enhancement layers at the coder, hierarchical coding can overall provide lossless compression of data at a higher resolution and / or a higher frame rate for a given transmission bitrate. Examples of hierarchical coding methods based on a hierarchy include LCEVC: MPEG-5 Part 2 LCEVC ("Low Complexity Enhancement Video Coding") and VC-6: SMPTE VC-6 ST-2117. The former is described in International Patent Application No. PCT / GB2020 / 050695 (and related standard documents) published as International Publication No. WO2020 / 188273, and the latter is described in International Patent Application No. PCT / GB2018 / 053552 (and associated standard documents) published as International Publication No. WO2019 / 111010. All of these documents are incorporated herein by reference. However, the concepts described herein need not be limited to these specific hierarchical coding methods.

[0151] A further example is described in International Publication No. WO2018 / 046940, which is also incorporated herein by reference. In this example, sets of residuals are encoded with respect to residuals stored in a temporal buffer.

[0152] LCEVC (Low Complexity Enhancement Video Coding) is a standardized coding method defined in a standard specification including the text of ISO / IEC 23094-2 Ed1 Low Complexity Enhancement Video Coding (issued in November 2021), which is also incorporated herein by reference.

[0153] Figures 4A and 4B schematically illustrate selected features of the LCEVC encoder 402 and the LCEVC decoder 404, showing how LCEVC can be used to efficiently encode and decode the still image region 3. Details of further embodiments for these types of encoders and decoders are described in previously published patent application GB1615265.4 and International Publication WO2020188273. These documents are hereby incorporated by reference into this specification.

[0154] In each of the encoder 402 and the decoder 404, items are shown at two logical levels. The two levels are separated by a dashed line. Items at the first, highest level relate to data at a relatively high quality level. Items at the second, lowest level relate to data at a relatively low quality level. The relatively high quality level and the relatively low quality level relate to a hierarchical structure having multiple quality levels. In some examples, the hierarchical structure includes more than two quality levels. In such examples, the encoder 402 and the decoder 404 may include more than two different levels. Above and / or below the levels shown in Figures 4A and 4B, there may be one or more other levels.

[0155] Referring to Figure 4A, the encoder 402 obtains input data 406 at a relatively high quality level. The input data 406 includes a first representation of a first time sample t1 of a signal at a relatively high quality level. The input data 406 may include, for example, an image generated by the image generator 31. The encoder 402 uses the input data 406 to derive downsampled data 412 at a relatively low quality level, for example, by performing a downsampling operation on the input data 406. When the downsampled data 412 is processed at a relatively low quality level, such processing generates processed data 413 at a relatively low quality level.

[0156] In some examples, generating the processed data 413 includes encoding the downsampled data 412. Encoding the downsampled data 412 generates an encoded signal at a relatively low quality level. The encoder 402 may output the encoded signal, for example, for transmission to the decoder 404. Instead of being generated at the encoder 402, the encoded signal may be generated by an encoding device separate from the encoder 402. The encoded signal may be an H.264 encoded signal. H.264 encoding can involve arranging an image sequence within a group of pictures (GOP). Each image within the GOP represents different temporal samples of the signal. A given image within the GOP may be encoded using one or more reference images associated with previous and / or subsequent temporal samples from the same GOP in a process known as "inter-frame prediction."

[0157] Generating the processed data 413 at a relatively low quality level may further involve decoding an encoded signal at a relatively low quality level. The decoding operation may be performed to emulate the decoding operation at the decoder 404, as will become apparent below. Decoding the encoded signal generates a decoded signal at a relatively low quality level. In some examples, the encoder 402 decodes the encoded signal at a relatively low quality level to generate a decoded signal at a relatively low quality level. In other examples, the encoder 402 receives a decoded signal at a relatively low quality level, for example, from an encoding and / or decoding device separate from the encoder 402. The encoded signal may be decoded using an H.264 decoder. As a result of H.264 decoding, an image sequence at a relatively low quality level (i.e., a sequence of temporal samples of the signal) is obtained. After the H.264 decoding process is complete, none of the individual images exhibit temporal correlation between different images in the sequence. Thus, any utilization of the temporal correlation between consecutive images used by H.264 encoding is removed during H.264 decoding because the consecutive images are separated from each other. Accordingly, subsequent processing is performed on a per-image basis as the encoder 402 processes the video signal data.

[0158] In one example, generating processed data 413 at a relatively low quality level further involves obtaining correction data based on a comparison between the downsampled data 412 and the decoded signal obtained by the encoder 402, for example, based on the difference between the downsampled data 412 and the decoded signal. The correction data can be used to correct errors introduced in the encoding and decoding of the downsampled data 412. In some examples, the encoder 402 outputs correction data (e.g., for transmission to the decoder 404) as well as the encoded signal. Thereby, the receiver can correct errors introduced in the encoding and decoding of the downsampled data 412.

[0159] In some examples, generating processed data 413 at a relatively low quality level further includes correcting the decoded signal using the correction data. In other examples, instead of using the correction data to correct the decoded signal, the encoder 402 uses the downsampled data 412.

[0160] In some examples, generating the processed data 413 involves performing one or more operations other than the encoding, decoding, obtaining, and correcting actions described above.

[0161] However, in some examples, no processing is performed on the downsampled data 412.

[0162] Derive upsampled data 414 at a relatively high quality level by using data at a relatively low quality level, for example, by performing an upsampling operation on the data at a relatively low quality level. The upsampled data 414 includes a second representation of the first time sample of a signal at a relatively high quality level. The coder 402 obtains a set of residual elements 416 that can be used to reconstruct the input data 406 using the upsampled data 414. The set of residual elements 416 is associated with the first time sample t1 of the signal. The set of residual elements 416 is obtained by comparing the input data 406 with the upsampled data 414.

[0163] In this example, the coder 402 generates a set of temporal correlation elements 426. The term "temporal correlation element" is used herein to mean a correlation element that indicates the degree of temporal correlation. The temporal correlation element can further be a spatio-temporal correlation element that indicates the degree of spatial correlation between the residual elements. In this example, the set of temporal correlation elements 426 is associated with both the first time sample t1 of the signal and the second time sample t0 of the signal. In the examples described herein, the second time sample t0 is a previous time sample with respect to the first time sample. However, in other examples, the second time sample t0 is a subsequent time sample with respect to the first time sample t1. In some examples, when the input data 406 includes a sequence of time samples, a previous time sample means a time sample that precedes the first time sample t1 in the input data. When the first time sample t1 and the previous time sample are arranged in the presentation order, the previous time sample precedes the first time sample t1.

[0164] The second time sample t0 can be the time sample immediately preceding the first time sample t1. In some examples, the second time sample t0 is a time sample that precedes the first time sample t1 but is not the time sample immediately preceding the first time sample t1.

[0165] In this example, the set of time correlation elements 426 indicates the degree of spatial correlation between multiple residual elements in the set of residual elements 416. The set of time correlation elements 426 also indicates the degree of temporal correlation between first reference data based on the input data 406 and second reference data based on a representation of, for example, a second time sample t0 of the signal at a relatively high quality level. Thus, the first reference data is associated with the first time sample t1 of the signal, and the second reference data is associated with the second time sample t0 of the signal. The first reference data and the second reference data are used as references or comparators to determine the degree of temporal correlation associated with the first time sample t1 of the signal and the second time sample t2 of the signal. The first reference data and / or the second reference data may be at a relatively high quality level.

[0166] In some examples, the first reference data and the second reference data include first and second sets of spatial correlation elements, respectively, the first set of spatial correlation elements being associated with a first time sample t1 of the signal and the second set of spatial correlation elements being associated with a second time sample t0 of the signal.

[0167] In another example, the first reference data and the second reference data comprise first and second representations of the signal, respectively, the first representation associated with a first time sample t1 of the signal and the second representation associated with a second time sample t0 of the signal.

[0168] The set of time correlation elements 426 is hereinafter referred to as "Δ t We call it the "correlation factor" because the time correlation is t This is because data from different time samples is used to generate the correlation element 426 .

[0169] In this example, the encoder 402 instead t Transmit the set of correlation elements 426. t The set of correlation elements 426 exploits the temporal redundancy present at higher residual levels, so that Δ tThe set of correlation elements 426 may be small where there is a strong temporal correlation and may include more correlation elements with zero values in some cases. Thus, when applied to the still image region 3 (which is static and changes only internally) as compared to the warpable images 2a, 2b, 2c (whose boundaries may change from frame to frame), less data is used to Δ t send the set of correlation elements 426.

[0170] Next, referring to FIG. 4B, the decoder 404 receives the data 420 based on the downsampled data 412 and Δ t receives the set of correlation elements 426.

[0171] If the encoder 402 processes the downsampled data 412 to generate the processed data 413, the decoder 404 processes the received data 420 to generate the processed data 422. The processing may include decoding the encoded signal to generate a decoded signal at a relatively low quality level. In some examples, the decoder 404 does not perform such processing on the received data 420. The upsampled data 414 is derived using data at a relatively low quality level (e.g., the received data 420 or the processed data 422). The upsampled data 414 may be derived by performing an upsampling operation on the data at a relatively low quality level.

[0172] The decoder 404 obtains a set of residual elements 416 based at least in part on the Δ t set of correlation elements 426. The set of residual elements 416 is available for reconstructing the input data 406 using the upsampled data 414.

[0173] In the present disclosure, implementations for integrating hybrid backward-compatible coding techniques with existing decoders are described, optionally via software updates. In a non-limiting example, the present disclosure relates to the implementation and integration of MPEG-5 Part 2 Low Complexity Enhanced Video Coding (LCEVC). LCEVC is a hybrid backward-compatible coding technique that combines different video coding formats, base codecs (i.e., codec pairs, such as AVC / H.264, HEVC / H.265, or any other current or future codec, as well as non-standard algorithms, such as VP9, AV1, and others) with one or more enhancement levels of encoded data, resulting in a flexible, adaptive, highly efficient, and low computational cost coding format.

[0174] In an example of hybrid backward-compatible coding techniques, a base stream is formed using a downsampled source signal encoded using a base codec. An enhancement stream is formed using a set of encoded residuals that correct or enhance the base stream, for example, by increasing the resolution or frame rate. There may be multiple levels of enhancement data within the hierarchical structure. In one arrangement, the base stream may be decoded by a hardware decoder, and the enhancement stream may be suitable for processing using a software implementation. Thus, the stream is considered to be a base stream and one or more enhancement streams, and usually two enhancement streams are possible, but often one enhancement stream is used. Notably, while the base stream may typically be decodable by a hardware decoder, the enhancement stream(s) may be suitable for software processing implementations with appropriate power consumption. The stream can also be considered as layers.

[0175] The combined intermediate image is then upsampled again to obtain a preliminary output image at the highest resolution. A second enhancement sublayer is combined with the preliminary output image to obtain a combined output image.

[0176] The second enhanced sublayer may be derived in part from a time buffer that is a store of the second enhanced sublayer used in the previous frame. By using the time buffer, the amount of data that needs to be included as part of the encoded frame is reduced. The time buffer may be used similarly for the first enhanced sublayer.

[0177] An indication of whether the time buffer can be used for the current frame or which part of the time buffer can be used for the current frame may be included in the encoded frame. Alternatively, the decoder itself may determine that the time buffer cannot be used for the current frame, such as when the first or second enhanced sublayer was dropped for the previous frame or received incorrectly, causing the time buffer to become stale.

[0178] Similarly, an indication of whether the time buffer should be used for the current frame may be sent to the encoder before encoding. For example, after the decoder fails to receive the first or second enhanced sublayer for the previous frame, the decoder may send an enhanced layer (or sublayer) drop indication. When the encoder receives the drop indication, the encoder may transmit a set of residual elements 416 instead of a set of delta t correlation elements 426.

[0179] Video frames are encoded hierarchically rather than using a block-based approach as done in MPEG family of algorithms. Encoding the frame hierarchically includes generating a residual for the full frame and then generating a reduced or subsampled frame, among other things. In the examples described herein, the residual may be considered the error or difference at a particular quality or resolution level.

[0180] For the purpose of context only, since the detailed structure of LCEVC is known and described in the approved draft standard specification, Figure 4B shows, assuming H.264 as the base codec, how LCEVC operates on the decoding side in a logic flow. A person skilled in the art will understand, based on the general description of LCEVC, whether the examples described herein are also applicable to other multi-layer coding schemes (for example, those using a base layer and an enhancement layer). The LCEVC decoder operates at the individual video frame level. It takes as input a low-resolution image decoded from a base (H.264 or other) video decoder and LCEVC enhancement data, and generates a decoded full-resolution image ready to be rendered to the display view. The LCEVC enhancement data is typically received either in the supplementary enhancement information (SEI) of the H.264 network abstraction layer (NAL) or in the packet identifier (PID) of additional data, and is separated from the base-encoded video by a demultiplexer. Therefore, the base video decoder receives the demultiplexed encoded base stream, and the LCEVC decoder receives the demultiplexed encoded enhancement stream. This enhancement stream is decoded by the LCEVC decoder to generate a set of residuals for combination with the low-resolution image decoded from the base video decoder.

[0181] LCEVC can be quickly implemented in existing decoders by software updates and is essentially backward compatible. This is because a device that has not yet been updated to decode LCEVC can play back video using the basic base codec, which further simplifies the introduction.

[0182] In such a context, this specification proposes a decoder implementation that integrates decoding and rendering with existing systems and devices that perform base decoding. The integration is easy to introduce. It also enables wide codec and player vendor support and can be easily updated to support future systems. Embodiments of the present invention particularly relate to how to implement LCEVC to provide decoding of protected content in a secure manner.

[0183] The proposed decoder implementation may be provided through a software library optimized for decoding MPEG-5 LCEVC enhancement streams and provides a simple but powerful control interface or API. This gives developers flexibility to deploy LCEVC at any level of the software stack, from low-level command-line tools to integration with commonly used open-source encoders and players. In particular, embodiments of the present invention generally relate to driver-level implementations and system-on-chip (SoC) level implementations.

[0184] The terms LCEVC and enhancement may be used interchangeably herein. For example, an enhancement layer may include one or more enhancement streams, i.e., residual data of LCEVC enhancement data.

[0185] FIG. 5 is a schematic diagram showing the process flow of LCEVC. In the first step, the base decoder decodes the base layer to obtain a low-resolution frame (i.e., the base layer). As the next step, the first enhancement (a sublayer of the enhancement layer) corrects the artifacts in the base. As a further step, the final frame (for output) is reconstructed at the target resolution by applying further (e.g., further residual details) sublayers of the enhancement layer. This shows that by making the most of the characteristics of existing codecs and enhancement, LCEVC improves quality and reduces the overall computational requirements for encoding. Embodiments of the present invention provide a way to achieve this in a secure manner (e.g., when dealing with protected content).

[0186] FIG. 6 shows the enhancement layer. As can be seen from the figure, the enhancement layer is sparse and contains very detailed information, and is not interesting (or of no value to the viewer) without the base video. However, the subtle movements in the rendered XR view will increase the information stored in this enhancement layer.

[0187] FIG. 7 shows a comparison of the latency when using a hierarchical coding scheme (such as LCEVC) and the latency when using a full serial coding scheme within a VR or XR pipeline.

[0188] As shown in FIG. 7, both methods start from rendering 710 in the source device. This can be any process that generates frame data such as image data or point cloud data, as described above.

[0189] In the full serial method, each frame is then encoded at full resolution (720). This can be implemented using any standard codec suitable for frame data, such as h.264, HEVC, AV1, or VVC for video data (usually a single-layer codec).

[0190] Next, in step 730, the encoded frame is transmitted from the source device to the destination device. For example, the encoded frame can be transmitted between nodes of a content rendering network or between a server and a user device.

[0191] In step 740, the encoded frame is received and stored in a jitter buffer until the receiving entity is in a state where it can process the frame.

[0192] In step 750, the receiving entity decodes the full-resolution frame.

[0193] The full serial mode ends with the reprojection 760 and display 770 of the frame data. The reprojection may include warping as described above.

[0194] The hierarchical coding mode is different in that after rendering 710, each frame is preprocessed (721) to generate a base layer. This typically includes downsampling such as a reduction in spatial resolution for the image data.

[0195] Next, in step 723, the base layer is encoded using a base codec. This can be implemented using any standard codec suitable for the frame data, such as h.264, HEVC, AV1, or VVC for video data (usually a single-layer codec). Alternatively, the base codec itself can be a hierarchical codec. The base codec may be irreversible, and when the base layer is encoded and then decoded, the original data of the base layer may not be exactly restored.

[0196] Finally, at step 725, the coder uses data from the encoded base layer to generate at least one enhancement layer. This typically involves a comparison between the original frame generated by rendering 710 and the decoding of the encoded base layer generated at step 723. For example, this can be implemented using the LCEVC technology described with respect to FIG. 4A. Step 725 may also include performing an encoding scheme on the enhancement layer to generate an encoded enhancement layer. This can include inter-frame encoding of the difference between the enhancement layer of the current frame and the enhancement layer of the previous frame. Additionally or alternatively, a general encoding scheme for data transmission, such as the addition of check bits, can be performed.

[0197] The hierarchical coding scheme also differs from the serial coding scheme in that the transmission and decoding can be split into two parallel processes or streams.

[0198] The first parallel process begins at step 731 where the encoded base layer is transmitted. This can begin as soon as the base layer is encoded at step 723. The receiving entity stores the encoded base layer of each frame in jitter buffer 741 until it is in a state to decode the base layer at step 751. Jitter buffer 741 can be made smaller than jitter buffer 740. This is because the base layer is a downsampled version of the frames stored in jitter buffer 740 of the serial coding scheme. Thus, the encoded frame(s) in jitter buffer 741 occupy less space and are decoded in less time (and / or with fewer computing resources) than the full-resolution frame(s) in jitter buffer 740.

[0199] The second parallel process begins with step 733 where the enhancement layer(s) is / are transmitted. This can begin after the enhancement layer(s) is / are generated and / or encoded at step 725. When the encoded enhancement layer is received at the destination device, it is decoded (753) to recover the enhancement layer.

[0200] Next, the first and second parallel processes are merged in a combining step 755. In the combination, the base layer and the enhancement layer are combined to produce a reconstructed representation of the original frame rendered at step 710. In the case of a reversible layer codec, the reconstructed representation is equivalent to the original frame, and in the case of an irreversible layer codec, the reconstructed representation should have at least the same characteristics as the original frame in terms of, for example, resolution. For example, the combination can include upsampling the base layer and using values from the enhancement layer to correct compression losses in the upsampled base layer caused by the base codec. The upsampling of the base layer can alternatively be incorporated at the end of the second parallel process, before the combining step 755.

[0201] The layered approach ends with reprojection 760 of the frame data and display 770. These steps are the same as in the case of the fully serial approach.

[0202] In the fully serial approach, since the frame is encoded and decoded at full resolution, it takes more time than encoding and decoding at a lower base resolution. Further, jitter buffering takes more time or requires more processing resources and more space at full resolution than at a lower base resolution.

[0203] On the one hand, the hierarchical coding method requires additional processes including pre - processing and post - analysis (generating enhancement layer(s)) in the encoder, and decoding of the enhancement layer and synthesis of the decoded layers in the decoder. Nevertheless, the time / resources saved by encoding and decoding at a lower base resolution outweigh the additional complexity of hierarchical coding. As a result, the overall latency is lower for hierarchical coding than for full - resolution coding, especially when the processing of the enhancement layer and the base layer is executed in parallel as much as possible.

[0204] Hierarchical coding also provides new degrees of freedom when coding and transmitting data. The enhancement data can be dropped in case of a sudden bandwidth reduction, reducing latency jitter. Furthermore, latency improvement can be achieved by using specific structures found in hierarchical coding including tiles and stripes (such as the LCEVC standard). More specifically, the base layer encoding of the frame region can be decoded in a series of parallel stripes. In implementations such as LCEVC, since only one stripe is required for the calculation of the enhancement layer, the calculation of the enhancement layer can start immediately once one stripe of the base layer has been decoded by the encoder. Similarly, the base layer can be transmitted as individual encoded stripes instead of a fully - encoded frame, enabling earlier start of the transmission of the base layer of the frame. Additionally, the enhancement layer enables encoding with a higher bit depth. For example, LCEVC supports a 14 - bit depth map and HDR even if the base encoder only supports an 8 - bit or 10 - bit base layer.

[0205] Figure 8 schematically illustrates VC-6 encoding. This is described in more detail in the standard document VC-6, SMPTE VC-6 ST-2117, and International Publication No. WO2019 / 111010. All of these documents are hereby incorporated by reference into this specification. Similar to LCEVC encoding, VC-6 uses downsampling to transmit the encoded base layer at a reduced resolution and restore the original data at the decoder using one or more enhancement layers.

[0206] Figure 9 schematically illustrates an example of a network system for distributed rendering. This may be referred to as a "content rendering network".

[0207] The first CRN node 910 performs first-pass volumetric rendering of the relevant zone of view within the virtual environment. This rendering is performed once for a plurality of viewing devices and includes the most computationally expensive parts of rendering such as ray tracing. This rendering may be performed at a relatively low frame rate (such as 25 fps) and may include motion information for upsampling at downstream nodes within the network.

[0208] The frame rendered by the first CRN node 910 is transmitted to the edge CRN node 920 closer to a particular user or particular user group. If there are multiple users or multiple user groups, the frame rendered by the first CRN node 910 may be transmitted to a plurality of edge CRN nodes 920 each associated with each user or the plurality of users.

[0209] Each edge CRN node 920 performs second-pass rendering of a user viewport (i.e., the area of the virtual environment visible to a particular user) to display the 3D environment as described above, and encodes a frame having one or more of an image layer and a depth map layer. This rendering can be performed once per user group, for example, if the data is in the MPEG Immersive Video (MIV) format defined in ISO / IEC FDIS 23090-12 (incorporated herein by reference). Alternatively, if the data is in another video format, the rendering can be performed once per user. This rendering can be performed at a higher frame rate (such as 50 fps) using interpolation based on the motion information generated by the first CRN node 910.

[0210] The frame rendered by the edge CRN node 920 is then streamed to the user device 940 via the network 930. The network 930 can have an air interface, such as between a user headset and a user base unit.

[0211] The user device 940 performs low-complexity decoding in terms of display resolution and display frame rate. At this stage, spatial warping reprojection can be applied using the depth map layer.

[0212] Even when the network speed is as low as 30 Mbps, the inventors have found that the aforementioned coding techniques can support streaming that is resilient to packet loss and can compensate for any network limitations by dropping layers, so that acceptable VR streaming is guaranteed as much as possible, and higher-resolution streaming can be obtained if supported by the network and the user device.

[0213] The following forms part of the description.

[0214] XR and the Metaverse: How to Achieve Interoperability, Wow Factor, and Mass Adoption Eliminate any inevitable hype and make it foolproof. In one form or another, the metaverse and extended reality (XR) are here to stay and will become the next generation of the internet. It is beneficial to understand them better, along with the corresponding requirements and technical implications.

[0215] However, despite the fact that affordable XR devices are becoming fit for purpose, the remaining bottlenecks to overcome are often the vast amounts of data required to enable an immersive XR experience of appropriate quality, for example, through "split computing" or "cloud rendering".

[0216] In this application, "split computing" is the process of dividing the computation across two or more devices and running the 3D rendering on a device different from the display device (near or far). The rendering can also be further split across multiple resources ("hybrid rendering"). The resulting rendered frames are then streamed ("cast") to the display device.

[0217] The inventors considered several technical challenges related to what happens before and after 3D rendering. · Three important user requirements for mass market adoption: lightweight devices, interoperability, and no compromise on quality. · Moore's Law and Koomey's Law cannot bridge the gap in graphics processing power between XR devices and gaming hardware. As a result, the inventors confirmed that high-quality, interoperable XR applications can instead operate via split computing and ultra-low latency video casting. · The problem of data compression for delivering volumetric objects to a rendering device and then casting high-quality ultra-low latency video to an XR display requires an efficient use of available resources, particularly a low-complexity multi-layer data coding scheme that is very suitable for 3D rendering engines. · Two recently standardized low-complexity multi-layer codecs (MPEG-5 LCEVC and SMPTE VC-6) have what is needed to enable high-quality metaverse / XR applications within realistic constraints. LCEVC enhances any video codec (e.g., h.264, HEVC, AV1, VVC) and enables the ultra-low latency delivery of high-quality XR video (or even video+depth) within strict wireless constraints (<30Mbps). Also, latency jitter is reduced because top-layer packets can be dropped during runtime when network bandwidth suddenly drops. By applying the VC-6 tool kit to point cloud compression, it becomes possible to mass-distribute photo-realistic 6DoF volumetric video at unprecedented quality, and has received great evaluations from both end-users and Hollywood studios. · XR rendering and video casting are 1:1 operations in many scenarios, and content is rendered for individual users. As a result, power consumption can increase linearly with users, unlike broadcast and video streaming. In that context, sustainability and energy consumption are important considerations, strengthening the case for low-complexity multi-layer coding. · The ability to appropriately address the issue of data volume and quickly leverage available approaches and standards is one of the important factors that promotes the speed at which the XR metaverse is brought to the public.

[0218] What exactly is the metaverse? The metaverse promises to (further) remove geographical barriers by providing new virtual freedoms for work, play, travel, and social interaction. But what exactly is it, and why now?

[0219] John Riccitiello (CEO of Unity) provided a sage definition of the metaverse (short for "meta-universe"). "It's the next generation of the Internet, always real-time, mostly 3D, mostly interactive, mostly social, mostly persistent." In fact, it's a new type of Internet user interface for interacting with other parties and accessing data as a seamless and intuitive extension of the physical world in front of us, inspired by the online 3D worlds already familiar to users of multiplayer video games. Metaverse solutions often aim to replicate the six degrees of freedom (6DoF) and inherent depth witnessed in the physical world, in contrast to the flat 2D interfaces we use to browse online. It is often described in the same sense as the term "extended reality" (XR, a combination of virtual reality and augmented reality), but the metaverse promises to be the biggest step change in the evolution of network computing since the introduction of the World Wide Web in the 1990s, which shifted from a text-based interface that was relatively accessible to anyone to the intuitive "browsing" of hypertext using multimedia components.

[0220] This new paradigm (metaverse, cyberspace, 6DoF 3D) is different and powerful in that it replicates the depth and intuitiveness inherent in the physical world, as opposed to the flat hypertext interfaces we currently use.

[0221] Why now? Gamers have been playing in 3D worlds with six degrees of freedom (6DoF) for decades, so it's natural to wonder why, even after 20 years, we still haven't seen 6DoF 3D e - shopping and / or working in Excel spreadsheets. Thanks to GTA V, we can freely roam a pretty good facsimile of Southern California, so why are Microsoft Office, Amazon, Instagram, and SAP all still so frustratingly flat?

[0222] In fact, migrating non - gaming applications to 6DoF 3D has been hampered by several technical barriers until now, but these are now being overcome.

[0223] Certainly, the obvious barrier to immersive XR has been represented by clunky VR headsets that couldn't meet the resolution and frame rate required for a smooth experience. Some people may still remember trying out a virtuality headset in the early 90s and feeling motion sick. Some may also remember comments about the awkward form factor or concerns that VR users "isolate" in the digital world. Commercially available headsets are now much lighter, enable non - tethered augmented reality, and finally function well enough visually to be appealing to the average consumer. The next headsets to be released will further address some of the remaining challenges to immersion. For example, projecting an image of the user's face to reduce the sense of "physical separation" from people outside the headset, adding sensors for real - time eye and face tracking to enable more realistic social interactions in the metaverse, and incorporating variable - focus display technology to address the focus problem (technically called "vergence - accommodation conflict", which means the display is forced to focus approximately 1.5m ahead regardless of depth). Overall, we are rapidly approaching headsets that are sufficient for mass adoption.

[0224] However, suboptimal headsets were not the only barrier to the rise of the metaverse. The 6DoF 3D digital world doesn't necessarily require immersive XR. This is evidenced by the fact that 3D video games have thrived on conventional flat screens for decades. Why don't we see office productivity tools, e-commerce websites, or other software applications following in the footsteps of video games?

[0225] Another major barrier to the non-gaming use of 3D worlds has been, until now, the difficulty of integrating intuitive 6DoF controls with the peripheral input devices commonly used on laptops, tablets, and mobile devices.

[0226] Keyboards, mice, touchpads, and touchscreens are great, but none are particularly well-suited for 6DoF navigation in 3D environments. Except for game controllers, the holy grail for mastering intuitive 6DoF controls has finally matured in recent years. Gesture controls, driven by inexpensive cameras and infrared proximity sensors, are now found in many devices.

[0227] This type of immersive interface is not limited to face-mounted XR displays. Even conventional flat-screen devices (e.g., laptops, mobiles, and large screens) are now equipped with cameras and IR sensors capable of gesture detection and head tracking. Already today, for just a few thousand dollars, you can buy a glasses-free autostereoscopic 8K screen. This tracks the movement of the head and eyes and projects a volumetric scene that appears to emerge from the screen, and the screen effectively acts like a window into a 6DoF 3D digital world.

[0228] For some applications, immersive 3D interfaces with integrated gesture control will evolve as revolutionary as hypertext and touchscreen interfaces have in the past few decades. Hypertext and traditional ("linear") video feeds will also continue to exist within the metaverse. They will simply be presented in the context of an immersive user interface and intuitively manipulated in 6DoF.

[0229] In fact, why not yet? There is a demand for more intuitive and immersive digital applications, and everything described is already technically possible today with relatively inexpensive devices. Many people are already starting to purchase these devices. Soon, more than 20 million households will be equipped with at least one of the latest generation of headsets. Lighter AR glasses can already perform augmented reality as described above, and a new and improved generation is expected soon, spurred by the entry of other big tech players into this market. Some autostereoscopic displays without glasses are close to the cost of traditional displays. Almost all flat screen devices are equipped with cameras and IR sensors capable of gesture tracking and head tracking.

[0230] There is a consumer demand for more intuitive and immersive digital applications, and all the technologies described are currently on the market and affordable, so why aren't we all already in the metaverse? What are the remaining barriers?

[0231] Key user requirements for mass market adoption In the jargon of the adoption cycle, XR consumers are moving from innovators (before 2020) to early adopters (2020 - 2022), and new XR experiences that attract people are emerging every day. Also, in the case of the proverbial "early majority" (which will necessarily incorporate non-gaming use cases as well), will everything function smoothly and seamlessly well enough to attract them?

[0232] To address mass adoption, it is necessary to meet three major user requirements. ·XR devices need to be small and light, ideally as light as glasses, and have a maximum peak power consumption of 1 - 2 watts. As a result, there is little room for electronics and batteries, and they are not suitable for processing high-quality 3D rendering in real time. A small form factor is an essential condition for scaling. Because people cannot "wear a gaming laptop on their face" (or even a high-performance mobile phone) for hours a day. ·Metaverse applications must be interoperable like web pages. That is, a very diverse range of viewing devices (including both XR headsets / smart glasses and more traditional TVs, mobile phones, tablets, or laptops) with very diverse computing capabilities must be able to access them. Interoperability is clearly important for service providers to ensure the largest possible user base for their services. · The quality of experience cannot be compromised. End users expect a visually beautiful, realistic, smooth, and immersive experience. In an average football game today, it is hardly surprising that video gamers take it for granted that they can see the beads of sweat dripping from the eyebrows of that virtual football player. While some imperfections may be tolerated in traditional 2D user interfaces, the point of the metaverse is precisely the illusion of "presence", and inevitably, first-class audio-visual quality, high-quality 3D objects, realistic lighting, high resolution, high frame rate, and low latency real-time are required. Sketchy objects and slow frame rates are not sufficient and can even make users feel queasy (about the movement). Naturally, the aforementioned McKinsey analysis shows a high correlation between the realism of the experience and the frequency of use, and companies are being pushed to build more realistic experiences.

[0233] After identifying the above requirements, the inventors confirmed that the first two requirements necessitate seamless "split computing". That is, most of the calculations (especially 3D rendering) need to be executed on another device (in some cases, even in the cloud), so that as a result, a more powerful GPU than what could otherwise be equipped can be time-shared as needed. The resulting rendered graphics frames must be streamed to the device as high-resolution, high-frame-rate, ultra-low-latency stereoscopic video. As we will see in the next section, the constraints related to XR casting are, especially when wireless connectivity (wi-fi or 5G) is involved, almost the worst nightmare imaginable for video coding. Fortunately, it will also be seen that there are standard solutions that make this possible within realistic constraints.

[0234] In addition to enabling the display of sophisticated graphics on low-power display devices, streaming video to a display device after remote computing is the best way to ensure interoperability with destinations in the metaverse, which is a second major requirement. Regardless of the available computing capabilities, any device can connect to a remote server and receive a video stream (e.g., mobile phones, TVs, autostereoscopic displays, XR headsets, and XR smart glasses). The server can also seamlessly handle the differences in the quality that each device can display and adapt the format and quality of the video stream according to the state of a particular user device and network. In a sense, the video can function as new HTML, and the metadata, enhancement layers, and auxiliary data channels can function in the same way as hypertext objects that enrich the baseline web page.

[0235] The third requirement means the opening of many new types of large-scale datasets that need to be efficiently exchanged within the metaverse, in addition to traditional audio, images, and video (more than 90% of current Internet traffic). Examples include immersive audio, point clouds of various natures, meshes, textures, stereoscopic video, video + depth maps for reprojection, etc. The decisive difference between metaverse destinations and video games is that volumetric objects may be difficult to pre-load and need to rely on real-time access. These large 3D assets need to be streamed in real time much more frequently than in the case of gaming.

[0236] Separation of rendering and display Access to the metaverse doesn't necessarily require an ultra-immersive XR headset exclusively. XR integrates headsets (over 23 million by 2023 according to IDC), lighter glasses-sized XR viewers, and glass-free autostereoscopic displays with billions of more conventional mobile phones, tablets, PCs, and TVs. Conventional flat screens display virtual destinations in the same way as is done for today's 3D games.

[0237] There's a catch. To experience real 3D graphics, hardcore gamers are equipped with sufficient graphics horsepower (read as "the latest generation of discrete GPUs with active cooling"), while the average mass-market XR user may only have a headset with 50 times less graphics power than a typical gaming PC and an iGPU-powered laptop with similarly disappointing graphics power.

[0238] In other words, since users won't accept "wearing a PlayStation 5 on their face" and not everyone can be assumed to be willing to buy a gaming PC, it's wise to design the metaverse experience to fit the minimum common denominator of compute and power.

[0239] That minimum common denominator (a lightweight XR device consuming 1 watt) is far from being able to perform real-time decoding of photorealistic volumetric objects and high frame rate 3D rendering at stereoscopic 4K resolution. There is no room for hope in physical laws and silicon technology. The latest generation of gaming GPUs are large and consume 150 - 300 watts of power to perform these tasks. Considering that an average mobile phone starts to heat up when consuming more than 4 watts, adding cooling fans and the weight of the battery, one can understand the reason for the joke about "wearing a PS5 on the face". As a result, there are mobile GPUs that cannot match the level of rendering quality expected by end users. Simple games like Beat Saber can be rendered, but probably cannot immerse users in a photorealistic 6DoF experience. Moreover, headsets with the capabilities of tier 1 mobile phones are still too large for the average user to accept wearing for hours a day.

[0240] There is a graphics power gap of over 50 times for lightweight devices compared to the visual quality experienced by hardcore gamers.

[0241] The latest mobile SoCs use 5-nanometer silicon technology in terms of transistor size, and long-term R&D suggests that 2 nanometers could be the physical limit of silicon. Since 1 nanometer means about 10 electrons, due to the quantum nature of electrons, the scale at which electrons can jump across silicon gates regardless of the applied voltage is being reached. Additionally, while shrinking transistors from 5nm to 2nm further, although transistor density increases, due to the quantum tunneling effect, the power consumption per transistor is unlikely to decrease further. According to TSMC, the transition from 7nm to 5nm has increased transistor density by 80%, but the improvement in computing performance per watt is only 15% as already known. This effect is also known as Koomey's Law, named after Jonathan Koomey, a professor at Stanford.

[0242] Unlike Moore's Law, which tracks the evolution of peak computing performance, Koomey's Law tracks the evolution of computing performance per watt. Unfortunately, Koomey's Law has measured a significant decline in the ability to perform more calculations for the same unit of power. From a 100-fold improvement per decade in the 1970s (doubling every 18 months), the first two decades of the 21st century reached approximately 15-fold per decade (doubling every 2.6 years), and currently, due to the aforementioned reasons in quantum physics, it has further stagnated.

[0243] As a result of this stagnation, it seems unlikely (and perhaps impossible) to achieve a 50-fold improvement in processing power efficiency in the next decade.

[0244] The processing power gap is also relevant to interoperability, which is one of the major user requirements. If metaverse applications are to be the "new web" in truth, then any XR device must ensure that it can properly reproduce it with seamless interoperability. How can we efficiently handle end-user devices that can have literally orders of magnitude difference in available graphics rendering capabilities? Since further improvements in silicon technology function as a placebo, the inventors' idea for a solution is based on mainframe and client-server prototypes. In the 1980s, a single mainframe computer could control real-time user interfaces for dozens or even hundreds of user terminals. This concept can be adapted to enable an immersive digital world with seamless interoperability. In the case of simple tasks, the mainframe could be a mobile phone that can be carried in a pocket, but as soon as serious rendering capabilities are required, it can seamlessly utilize the GPU resources of the nearest available shared computing node.

[0245] Performing rendering externally to the XR device also means streaming volumetric objects to a rendering device that is more likely to be properly connected to the Internet, especially in the case of within a data center. The XR device (wirelessly connected to the Internet via Wi-fi or 5G, often <50Mbps due to distance from repeaters, obstacles, concurrent processing, and packet loss due to interference) only requires sufficient connectivity to process video streams.

[0246] In the case of a metaverse application that demands a great deal of 3D graphics, it is inevitable that rendering and display need to be separated. A rendering device capable of both sufficient power cranking and consumption (whether it be a handheld device, a nearby PC, or some powerful server somewhere "in the cloud") receives the compressed virtual objects, decodes them, runs the application, and renders the viewport (i.e., what the end user always sees). Even rendering itself can actually be split across separate stages on different machines. For example, a more powerful GPU can calculate lighting information (in some cases, even using costly algorithms such as ray tracing or path tracing, which require computing power more than 1,000 times that of general game engine lighting), while a less powerful nearby GPU can calculate the final rendering. In other cases, in complex 3D scenes with many users, costly lighting calculations may be "pooled" for multiple users, intermediate volumetric data is generated, and sent to multiple local edge nodes to render the final viewport for each individual user. Thus, rendering servers can be organized in a dynamic hierarchical structure, much like how cache servers in a content delivery network enable large-scale web content delivery. Such a metaverse infrastructure can be referred to as a "Content Rendering Network" (CRN). The resulting rendered viewport is then cast to a lightweight XR display device via ultra-low latency video streaming. Therefore, the XR device only needs to manage its sensors, process / send data on what the user is doing, and decode the received high frame rate, high resolution video.

[0247] This can actually be run at photo-realistic quality within 1 watt, particularly by using a low-complexity video compression scheme that fits within bandwidth and processing constraints.

[0248] Challenges of Distributed Computing for Enabling Large-Scale XR Regarding the large-scale introduction of XR, the importance of appropriately processing an enormous amount of data cannot be underestimated. Data compression and operations that conform to the constraints of robust up-to-date network connectivity quality, bandwidth, processing, and latency are what the metaverse game is all about, and there are three major technical challenges that the inventors are working on.

[0249] 1. Appropriate compression and streaming of volumetric objects to and between rendering devices require a new coding approach. Efficient compression / restore of 3D objects using appropriate irreversible coding techniques can enhance the possibility of effectively streaming point clouds, textures, and meshes. In the case of use cases of hybrid rendering, intermediate by-products are preferably efficiently transmitted between different nodes of the content rendering network. So far, most of the monolithic downloads sometimes done in the gaming world have been coded reversibly. A flexible software-based layered coding scheme suitable for efficient super-parallel execution is advantageous. 2. Perform ultra-low latency video encoding with a bandwidth of less than 30 - 50 Mbps not only for 4K at 72fps and above, but also for realistic Wi-fi / 5G sustained rates. For the experience not to be "jerky", it is desirable that the latency of video casting is low and stable. Some latency can be managed by rendering the viewport according to the prediction of where the user is looking after the estimated system delay, and then performing a last-millisecond reprojection according to where the user is effectively looking. However, it is still about 20 - 30 milliseconds, and the processing constraints are severe. At such ultra-low latency, many of the latest video compression efficiency tools cannot be used, so the bandwidth required to transmit a given quality increases. At the same time, wireless distribution means severe bandwidth constraints (<50Mbps) before packet loss starts to generate unacceptable latency jitter. In addition to selecting an appropriate protocol (with some forward error correction if possible), it is particularly useful to combine the most efficient video codec available with a compression enhancement scheme of low complexity that can reduce the bitrate and generate layers of data that can be discarded on the fly in case of sudden congestion. Note that an example of such a method is already available in the MPEG-5 LCEVC (Low Complexity Enhanced Video Coding) coding enhancement standard. This standard improves compression efficiency, reduces overall processing requirements, and generates layers of enhancement data that can be dropped in case of sudden network congestion without impairing the decoding of the base layer video. Therefore, the risk of latency jitter is reduced. 3. A powerful network backbone / CDN / wireless is between the rendering device and the XR display device, and sufficient cloud rendering resources are available (e.g., "CRN"). In split computing, if the network and shared resources are not robust, the inalienable quality of experience can be at risk. Telecommunications carriers, cloud providers, and CDNs must work extra hours to ensure that as many people as possible are in the state emphasized in Point 2 above, that is, that users can access a powerful GPU server within a few milliseconds from a network ping and have access to an end-to-end reliable connection of at least 30-50 Mbps. Currently, less than 10% of the population in developed countries subscribes to FTTH-grade connectivity and has introduced the latest generation of Wi-fi routers. In addition, only a small part of the GPU server infrastructure available in the cloud can be supported.

[0250] In the case of low-complexity layered coding LCEVC is the new hybrid multi-layer coding enhancement standard of ISO-MPEG. It is codec-independent in that it combines a low-resolution base layer encoded by some conventional codec (e.g., h.264, HEVC, VP9, AV1, or VVC) with a layer of residual data that reconstructs the full resolution. The LCEVC coding tool is particularly suitable for efficiently compressing details from both the processing and compression perspectives. On the other hand, by utilizing the conventional codec at low resolution, the hardware acceleration available for that codec is effectively used to make it more efficient.

[0251] LCEVC has been proposed as an important enabler for ultra-low latency XR streaming and offers the following advantages. 1. Significant compression advantages for all available hardware codecs (h.264, HEVC, or AV1). Since the LCEVC compression enhancement tool kit mainly operates within the spatial domain, its advantages are very effective for ultra-low latency coding where most temporal compression techniques (such as the pyramid of bi-predicted references that utilize subsequent frames) cannot be used. In addition to the important improvement in efficiency over non-enhanced coding (usually over 30 - 40%), it is crucial that with LCEVC, the absolute bandwidth can be kept within the constraints. LCEVC enables high frame rate stereoscopic 4K within 25 - 50 mbps, making high-quality wireless casting and cloud XR streaming much more feasible. The subjective quality difference around 30 Mbps often means the difference between "unwatchable" and "fit for purpose". 2. Unique latency jitter reduction due to the multi-layer structure inherent in LCEVC. LCEVC data can be transmitted in a separate lower-priority data channel and can be dropped during execution (even in the middle of transmitting encoded frames) in case of network congestion, without damaging the base layer and minimizing the impact on visual quality (i.e., the picture quality looks softer for a few frames, but there is no obvious spatial correlation impairment). This can immediately handle the frequent fluctuations and packet losses inherent in wireless transmission, avoiding the accumulation of latency jitter of tens of milliseconds every time the bandwidth suddenly drops. Furthermore, from the perspective of the overall end-to-end system latency, LCEVC also enables the transmission and decoding of the base layer while encoding the enhancement layer, obtaining additional degrees of freedom for reducing the encoding latency. 3. Due to the low processing complexity, LCEVC can be immediately used with efficient heterogeneous processing (e.g., 4K, 72fps with limited resource overhead for native hardware encoding / decoding). In the case of dedicated silicon acceleration where IP blocks are already available, extremely high resolution, frame rate, and bit depth (e.g., 16K, 120fps, 14 bits) can be efficiently encoded / decoded with a very small silicon footprint, enabling wireless streaming quality of retinal display XR on lightweight devices. 4. High bit-depth high dynamic range (HDR) HDR is important for a realistic immersive experience, and from previous analysis, the higher the bit depth, the better. Since LCEVC utilizes a 16-bit accelerated pipeline, it can encode HDR video at 12 bits or 14 bits without any processing and / or bitrate overhead while still using the available 8-bit or 10-bit base layer codec. The ability to send a 5.14-bit depth map along with the video enables more realistic depth-based reprojection and gaze-tracking-based variable focus adjustment, resulting in a better overall visual experience. Without a depth map, the reprojection performed just before display (to compensate for the difference between the actual viewport and the predicted viewport during rendering) is done by "flat" reprojection. This is one of the main visual impairments caused by system latency. The availability of the depth map is also useful in variable focus headsets that are close to gaze tracking, enabling the local device to determine the depth of the VR object the user is looking at and adjust the focal distance of the display accordingly to avoid vergence-accommodation mismatches (i.e., a major step towards visual realism). LCEVC's fast software-based coding tools (executed via a 16-bit accelerated pipeline) can effectively deliver the depth map along with the video whenever sufficient resources and / or bandwidth are available. In addition to making video + depth practical from a processing perspective, the overall bandwidth reduction enabled by LCEVC is also important to enable the transmission of the depth map within the constraints of the available bandwidth.

[0252] Importantly, the aforementioned advantages can be achieved while minimizing average system latency. LCEVC can structure the end-to-end pipeline to obtain maximum parallel execution, as shown in Figure 7, because it natively divides the computation into separate independent processes (similar to a kind of "vertical striping") without sacrificing compression efficiency.

[0253] The proper ultra-low latency implementation of LCEVC enhancement casting can thus produce a useful combination of enhanced visual quality, minimal average latency, and reduced latency jitter, making a case for local and / or cloud-based split computing with "retinal quality" 120fps wireless XR streaming to lightweight devices. It is an example of what is called "breathing life into the digital future", enabling creative and digital businesses to similarly design a whole new world of impactful experiences and user interfaces.

[0254] Speaking more specifically about LCEVC's multi-layer and low-complexity coding toolkit, SMPTE VC-6 (SMPTE ST2117-1) is another coding standard particularly suitable for the metaverse dataset due to its recursive multi-layer approach, use of S-trees, and its extensible coding toolkit (which uniquely includes super-parallel entropy coding and in-loop neural networks). The concept of VC-6 is shown in Figure 8.

[0255] In addition to applications in image / texture compression, professional video workflows, and AI media indexing acceleration, a variant of VC-6 for point cloud compression was used to compress the PresenZ volumetric video format, which is the only media format capable of rendering photo-realistic 6DoF volumetric experiences. This is described in International Publication No. WO2023 / 047119, which is hereby incorporated by reference into this specification. Some examples of media that benefit from this compression are available on the Steam store using the PresenZ player. Uniquely, the codec demonstrated ultra-fast software decompression by not only compressing huge datasets (over 50 gigabits per second) for delivery to end-users, but also decompressing them in real-time while leaving sufficient free processing resources for high-resolution, high frame-rate real-time rendering.

[0256] In the metaverse, the decompression of 3D objects, graphics rendering, and viewport compression are all closely related. In such situations, a hierarchical data structure is particularly advantageous. When using a hierarchical data structure, higher-quality textures or polygon models can be obtained from lower-quality versions and some data specifying the details can be added. As the object is approached, more data is fetched. Additionally, when decompression is ultra-fast and super-parallel, it may be desirable to operate directly on the compressed data rather than the uncompressed data to maximize the level of realism that can be processed with a given memory bandwidth.

[0257] In addition to these advantages, the hierarchical multi-layer signal format can also assist the world of AI-based indexers and make them more accurate and faster in classification tasks. This is a great advantage for metaverse bots. Low-quality signal representations enable the rapid detection of areas requiring further classification, and high-resolution decoding of regions of interest can be completed only for areas of particular interest. Multiple classifiers can perform different operations in parallel on the same compressed dataset, potentially distributed across multiple nodes.

[0258] Data types may increase in the metaverse. In certain applications, unique combinations of point clouds, meshes, auxiliary data for physical rendering engines, textures, and enhanced videos may be required. These require progressive and partial decoding of regions of interest. It may be necessary to process data with variable bit depths (in some cases higher than 10 bits), such as depth maps or point cloud coordinates.

[0259] Low-complexity multi-layer coding, which can efficiently execute software on general-purpose parallel hardware, has been proposed as a scalable solution interoperable with various devices having different display screens and processing capabilities. Flexibility is desirable, for example, to enable the same coding scheme to be executed as an ASIC / FPGA with a very small number of gates or as software on general-purpose graphics hardware with low processing power consumption.

[0260] Such design criteria are central to the video standards MPEG-5 LCEVC and SMPTE VC-6, and the MPEG Immersive Video ("MIV") and volumetric data compression projects within MPEG may also be useful for addressing some of these issues. Appropriately leveraging these last coding standards can enable high-quality XR to be achievable and interoperable.

[0261] It is also necessary to lay fiber optic cables, roll out 5G, and install new data centers, but that alone won't solve the problem. All available technologies must be used.

[0262] Higher quality with less energy Sustainability is also something to consider. In short, it is necessary to attract people and have an XR workflow without interruption, and to minimize energy costs.

[0263] It is very difficult to come up with reliable figures, and laudable ideas such as the greening of streaming are emerging to make it clearer what and how to measure. That being said, the impact on the media delivery environment is certainly relevant and rapidly expanding. Approximately 2% of the world's greenhouse gas emissions come from data centers. Data centers use about 200 terawatt-hours of electricity, which is comparable to the world's aviation industry. Taking this energy consumption, adding the remaining delivery elements of video streaming, considering an annual growth rate of 50% in usage, and further bearing in mind that Moore's Law is no longer very useful in terms of power consumption per processing unit, and adding the metaverse to the mix... the numbers are about to become a two-digit percentage of the world's total energy consumption.

[0264] Since XR rendering and video casting are often 1:1 operations, unlike broadcasting and video streaming, the power consumption may increase linearly according to the user. The low-processing approach is considered important for the overall sustainability of the metaverse, and another brick will be added to build a case for low-complexity multi-layered coding.

[0265] Realization of the metaverse The ever-expanding diverse 3D data volume and the stringent video streaming constraints of distributed computing are important issues to be addressed to ensure that immersive worlds can be provided to end-users in an interoperable and large-scale manner.

[0266] As highlighted above, the rapid adoption of available multi-layer coding standards and the development of new ones from the same IP / toolkit can greatly accelerate the development of high-quality, interoperable metaverse destinations.

[0267] The ability to appropriately address the data volume issue and quickly utilize available approaches and standards is one of the important factors in accelerating the speed at which the XR metaverse is brought to the masses.

Claims

1. A network system for generating a sequence of frames for rendering a dynamic 3D scene, wherein the system includes a first rendering node and a second rendering node. The first rendering node, Generate a first partially rendered sequence of frames, Layered encoding is performed on each of the first partially rendered frames to generate a sequence of encoded first partially rendered frames. The encoded sequence of first partially rendered frames is configured to be sent to the second rendering node. The second rendering node described above, The sequence of the encoded first partially rendered frames is obtained from the first rendering node. A network system configured to generate a second partially or fully rendered sequence of frames based on the first partially rendered sequence of encoded frames.

2. The network system according to claim 1, wherein the second rendering node is configured to decode the encoded first partially rendered sequence of frames to obtain the first partially rendered sequence of frames.

3. The network system according to claim 1, wherein the second rendering node is configured to perform layered coding on each second partially or fully rendered frame to generate a sequence of coded second partially or fully rendered frames.

4. The network system according to claim 3, wherein the first rendering node is configured to perform layered coding according to a first coding scheme, and the second rendering node is configured to perform layered coding according to a second coding scheme, wherein the first coding scheme is different from the second coding scheme.

5. The network system according to claim 1, wherein the frame rate for the second partially or fully rendered sequence of frames is greater than the frame rate for the first partially rendered sequence of frames.

6. The network system according to claim 1, comprising a display device, wherein the communication delay between the first rendering node and the display device is greater than the communication delay between the second rendering node and the display device.

7. The first rendering node is configured to obtain the view position from the display device before generating the first partially rendered sequence of frames. The network system according to claim 6, wherein the second rendering node is configured to obtain updated viewing positions from the display device before generating the second partially or fully rendered sequence of frames.

8. The network system according to claim 1, wherein generating the first partially rendered sequence of frames requires more processing resources than generating the second partially or fully rendered sequence of frames.

9. The network system according to claim 1, wherein each frame includes image data and depth map data, and the system includes a third rendering node configured to generate a third fully rendered sequence of frames by performing time warping and / or depth correction on the second partially or fully rendered sequence of frames using the depth map data.

10. The network system according to claim 1, wherein each frame includes point cloud data in which each of a plurality of points has a 3D position and one or more attributes.

11. The network system according to claim 10, wherein the second or third rendering node is configured to calculate depth map data based on the 3D position of a point.

12. The first rendering node is configured to generate one or more sequences of first partially rendered frames for a first number of users or display devices. The network system according to claim 1, wherein the second rendering node is configured to generate one or more sequences of a second partially or fully rendered frame for a second number of users or display devices, the second number being smaller than the first number.

13. The network system dynamically selects how many nodes and which specific nodes to use for the rendering process in response to at least one metric, A metric based on the complexity of the rendering task to be performed, Metrics based on available free space at each node, Metrics based on the location of the display device relative to the node network, A metric based on round-trip latency between the node and the display device, Metrics based on bandwidth available between nodes and from nodes to display devices, The network system according to claim 1, comprising a metric based on the number of different display devices requesting rendering of the same 3D scene within a viewpoint range.

14. Partially rendered frames, Point cloud data, Mesh data, The network system according to claim 1, which is encoded as volumetric data including one or more texture data.

15. The network system according to claim 1, wherein a partially rendered frame includes light field data.

16. The network system according to claim 1, wherein the partially rendered frames include spatial properties that enable the calculation of the behavior of sound in the 3D space.

17. The network system according to claim 1, wherein the partially rendered frames are encoded using at least a partially lossy encoding scheme.

18. The network system according to claim 1, wherein partially rendered frames are encoded at least partially using a layered encoding scheme.

19. The network system according to claim 1, wherein subsequent nodes receive only a portion of the data encoded by the first rendering node in response to specific locations of one or more viewpoints of the fully rendered frames to be computed.

20. The network system according to claim 1, wherein a subsequent node decodes only a subset of the encoded data generated by the first rendering node and received by the subsequent node that is necessary to fully render a particular field of view that it is rendering at any one point.

21. The network system according to claim 1, wherein partially rendered frames are encoded by at least partially using a point cloud format that represents points according to one or more coordinate systems, and each point is assigned one or more data attributes that specify visual characteristics including one or more of size, normal vector, motion information, color information, and transparency.

22. A method for encoding a sequence of frames representing a dynamic 3D scene, wherein each frame can consist of base layer image data and enhancement data, the method comprising performing layered encoding on the frames of the sequence of frames to generate an encoded frame having one or more of a base image layer and an enhancement image layer.

23. The method according to claim 22, wherein the enhanced image layer includes data used by the decoder device to reconstruct a higher resolution representation of the sequence of frames.

24. The method according to claim 22, wherein the enhanced image layer includes data used by the decoder device to reconstruct a higher bit depth representation of the sequence of frames with respect to the bit depth of the base image layer.

25. The method according to claim 22, wherein the enhanced image layer includes data used by the decoder device to reconstruct the distance of an object in the image from the viewer.

26. The method according to claim 22, wherein the enhanced image layer includes data used by the decoder device to reconstruct haptic feedback for the user.

27. The method according to claim 22, further comprising: in response to a decrease in transmit channel bandwidth, discarding enhanced data during transmission of the sequence of frames, indicating to the encoder that the enhanced data has been dropped; if the enhanced data has been dropped, refreshing the time buffer for the enhanced data encoding; and performing an instantaneous decoder refresh (IDR) on the enhanced data (but not necessarily on the base layer data) to take into account that the decoder has missed some of the previous enhanced data.

28. The method according to claim 22, wherein the layered coding scheme used is MPEG-5 LCEVC (Low Complexity Enhanced Video Coding) coding or SMPTE VC-6 coding.

29. The method according to claim 28, wherein at least one of the enhancement data is transmitted as embedded user data within the coefficients of the LCEVC data.

30. The method according to claim 29, wherein one or more residual coefficients of an image include embedded depth information representing the depth of a corresponding object relative to a viewpoint, and the decoder processes the embedded data to reconstruct a depth map associated with the image frame based at least in part on the embedded data.

31. The method according to claim 30, wherein the depth map is reconstructed by processing both the embedded data and the image data.

32. The method according to claim 22, wherein the frame-plus-depth data transmitted at a given frame rate is used by a display device to increase the frame rate via depth-based reprojection to match the display frame rate.

33. Each frame contains image data and depth map data. The method according to claim 22, comprising performing layered coding on a frame in the sequence of frames to generate an encoded frame having one or more of a base depth map layer and an enhanced depth map layer.

34. The method according to claim 33, wherein each frame includes an image layer and a depth map layer, and the encoded frame includes one or more of the base image layer, the base depth map layer, the enhanced image layer, and the enhanced depth map layer.

35. The method according to claim 33, wherein each frame includes the depth map data embedded in the image data, the base depth map layer is a base image layer having the embedded depth map data, and the enhanced depth map layer is an enhanced image layer having the embedded depth map data.

36. Receiving a depth map drop indicator that shows whether or not the depth map data has been dropped when transmitting the sequence of frames, The method according to claim 33, further comprising: discarding the depth map data of a frame if the depth map data is dropped; and performing layered coding on the image data of the frame to generate an encoded frame including a base image layer and an enhanced image layer.

37. Receiving frame-plus-depth data at a given frame rate, encoded using a layered coding scheme, A method for increasing the frame rate in a display device via depth-based reprojection using the depth data to match the display frame rate.

38. A bit sequence representing the encoding of a sequence of frames representing a dynamic 3D scene, wherein the bit sequence is Encoded data for the base depth map layer for the frame, A bit sequence comprising one or more of the following: encoded data for the enhanced depth map layer for the frame; and

39. A method for transmitting a sequence of frames representing a dynamic 3D scene, wherein each frame includes image data and depth map data, and the method Obtain an encoded frame containing one or more of the base depth map layer and the enhanced depth map layer, When transmitting the sequence of frames, determine whether the depth map data should be dropped, If the aforementioned depth map data should be dropped, Discard the depth map data from the encoded frame and obtain a reduced encoded frame that includes one or more of the base image layer and the enhanced image layer. Transmitting the reduced encoded frame, Otherwise, a method comprising transmitting the encoded frame.

40. A method performed by an encoder for encoding a sequence of frames, wherein the method is Layered coding is performed on the first frame of the sequence of frames to generate a first coded frame including a base layer and an enhancement layer. To use in the temporal encoding of subsequent frames, the components of the enhancement layer of the first encoded frame are stored in a time buffer, Sending the first encoded frame to the transmitter for transmission, Receiving an enhanced drop indicator that shows whether the enhanced layer was dropped when transmitting the first encoded frame, The process includes performing layered coding on a second frame of the sequence of frames to generate a second coded frame including a base layer and an enhancement layer, If the enhanced drop indicator indicates that the enhanced layer was not dropped, the enhanced layer of the second encoded frame is generated by referring to the time buffer. A method wherein, if the enhanced drop indicator indicates that the enhanced layer has been dropped, the enhanced layer of the second encoded frame is generated without referring to the time buffer.

Images