Augmenting a video or external environment with 3D graphics

Abstract

A rendering device for rendering a scene is provided, where the rendering device is configured to display a video object or allow an external environment to visually pass through to a viewer and render the scene as an augmentation of the video object or the external environment. The scene can be rendered in a hybrid manner, involving local rendering of a portion of the scene by the rendering device and remote rendering of another portion of the scene by a remote rendering system. Both the rendering device and the remote rendering system can have access to scene descriptor data, which can indicate changes in state of the scene over time. In this way, the rendering client can request remote rendering of a portion of the scene at a particular time state, e.g., accounting for network latency and / or processing latency.

Description

Technical Field

[0001] This invention relates to a computer-implemented method for rendering a scene using a rendering device, wherein the rendering device is configured to display a video object or allow an external environment to be visually conveyed to a viewer and render the scene as an enhancement of the video object or the external environment. The invention further relates to the rendering device and a remote rendering system and computer-implemented method for rendering portions of a scene using the rendering device. The invention further relates to a computer-readable medium comprising data for causing a processor system to perform any of the computer-implemented methods. Background Technology

[0002] In augmented reality (AR) or mixed reality (MR), computer-graphics-based 3D objects can be combined with the physical reality perceived by the viewer to enhance that physical reality. For example, a head-mounted AR or MR rendering device can visually convey the external environment to the viewer, for instance, via transparent or semi-transparent portions of the rendering device, and can be configured to render one or more computer-graphics-based 3D objects of a scene and display these objects in a way that allows them to be overlaid, mixed, blended, or otherwise visually integrated with the external environment from the viewer's perspective. Thus, the external world can be enhanced using digital content such as informational content (e.g., navigation directions) or entertainment (e.g., game characters).

[0003] Such computer graphics-based 3D objects (hereafter referred to simply as '3D graphics objects') can also be rendered to enhance video-based representations of physical reality. For example, head-mounted AR / MR rendering devices can include forward-facing cameras and can record and display the recorded video to viewers in real-time or near real-time, while simultaneously enhancing the video using rendered 3D graphics objects. Moreover, 3D graphics objects can be used to enhance other types of video. For example, in virtual reality (VR), one or more 3D graphics objects can be used to enhance so-called panoramic or omnidirectional video.

[0004] AR, MR, or VR are also collectively referred to as Extended Reality (XR).

[0005] It is known to perform rendering of 3D graphics objects on a user's rendering device for XR enhancement; this rendering is also referred to below as 'local' rendering. It is also known to perform rendering on a remote rendering system; this rendering is also referred to below as 'remote' rendering.

[0006] For example, [1] describes the viewport of a scene rendered entirely by an XR server in section 6.2.4. The XR server reportedly generates XR media on the fly based on incoming tracking and sensor information (e.g., a game engine). The generated XR media is provided for the viewport in a 2D (flattened) format, encoded, and transmitted over a 5G network. The tracking and sensor information is transmitted in the opposite direction. In the XR device, the media decoder decodes the media and renders the viewport directly without using viewport information.

[0007] Remote rendering can solve some of the problems associated with local rendering. For example, if the scene to be rendered is too complex (e.g., the required rendering performance exceeds the existing rendering capabilities) or if the rendering device is battery-powered and the battery is low, it may be desirable to render the scene remotely rather than locally. However, in other cases—for example, if the network connection is poor or if (ultra) low latency rendering of objects is desired—remote rendering can be disadvantageous.

[0008] Scenes can also be rendered partly by a remote rendering system and partly by a local rendering device, where such rendering is referred to as 'blended' or 'split' rendering. For example, [2] describes rendering more complex content, such as computer-aided design models, on a remote computing system, and streaming the rendered models to a client device that performs the generation and / or display of the models. The client device can enhance the remotely rendered content with locally rendered content, such as lightweight, low-latency content (e.g., user interface / user experience elements, ink, content related to real-world objects such as articulated hands, etc.). The client device can also perform depth-corrected blending to provide occlusion relationships between the remotely rendered content and the locally rendered content.

[0009] However, aside from the fact that individual objects are manipulable,[2] assuming the scene to be rendered is largely static, in which case the client device can manipulate the scene by notifying the remote rendering system using the Scene Graphics API. This allows the remote rendering system to be notified of the animation or movement of individual objects. However, this approach does not scale well to more dynamic scenes, such as those involving more complex animations or movements, or the appearance of new objects, because the client device would need to continuously notify the remote rendering system of any manipulation of individual objects, which could be computationally and network-intensive for the client device.

[0010] Another problem with the method in [2] is its sensitivity to latency, because manipulation of objects must first be detected at the client device, data representing the manipulation must be generated and uploaded to the remote rendering system via the scene graphics API, and remote rendering only begins after all the data has been received. If the data representing the manipulation is quite large, the upload may therefore take a long time, especially when the bandwidth to the remote rendering system is limited.

[0011] References

[0012] [1] Extended Reality (XR) in 5G, ETSI TR 26928V16.1.0

[0013] [2] Hybrid Rendering, WO 2020 / 209962 A1 Summary of the Invention

[0014] It is hoped that hybrid rendering of dynamic scenes can be achieved, thereby solving one or more of the above problems.

[0015] In a first aspect of the invention, a computer-implemented method is provided for rendering a scene using a rendering device. The rendering device can be configured to display a video object or allow an external environment to be visually conveyed to a viewer and render the scene as the video object or an enhancement of the external environment. The method may include:

[0016] Obtain scene descriptor data, wherein the scene descriptor data can identify a set of 3D graphical objects representing at least a portion of the scene, and wherein the scene descriptor data can indicate the change of the state of the scene over time;

[0017] Based at least on the scene descriptor data, determine the first part of the scene to be rendered locally by the rendering device and the second part of the scene to be rendered remotely;

[0018] The request is made to a remote rendering system authorized to access the scene descriptor data to render a second part of the scene to obtain pre-rendered scene data, which includes a pre-rendered version of the second part of the scene.

[0019] The request may include providing a time stamp to the remote rendering system in order to request the remote rendering system to render a second part of the scene in a state corresponding to the time stamp;

[0020] Obtain the pre-rendered scene data from the remote rendering system;

[0021] The scene is rendered as an enhancement of the video object or the external environment, wherein the rendering may include a first part of the scene rendered locally and a pre-rendered version of a second part of the scene.

[0022] In another aspect of the invention, a rendering apparatus is provided. This rendering apparatus can be configured to display video objects or allow an external environment to be visually conveyed to a viewer and to render a scene including 3D graphical objects as an enhancement of the video object or the external environment. The rendering apparatus may include:

[0023] Network interface with the network;

[0024] A data storage device including scene descriptor data, wherein the scene descriptor data can identify a set of 3D graphical objects representing at least a portion of the scene, and wherein the scene descriptor data can indicate the change of the state of the scene over time;

[0025] The processor subsystem can be configured to:

[0026] -Based at least on the scene descriptor data, determine the first part of the scene to be rendered locally by the rendering device and the second part of the scene to be rendered remotely;

[0027] -Through this network interface, a request is made to a remote rendering system authorized to access the scene descriptor data to render a second part of the scene to obtain pre-rendered scene data, which includes a pre-rendered version of the second part of the scene.

[0028] The request may include providing a time stamp to the remote rendering system in order to request the remote rendering system to render a second part of the scene in a state corresponding to the time stamp;

[0029] - Receive the pre-rendered scene data from the remote rendering system via the network interface;

[0030] - Render the scene as an enhancement of the video object or the external environment, wherein the rendering may include a first part of the scene rendered locally and a pre-rendered version of a second part of the scene.

[0031] In another aspect of the invention, a computer-implemented method is provided for rendering a portion of a scene for a rendering device. The rendering device can be configured to display a video object or allow an external environment to be visually conveyed to a viewer and render the scene as the video object or an enhancement of the external environment. The method may include:

[0032] Obtain scene descriptor data, wherein the scene descriptor data can identify a set of 3D graphical objects representing at least a portion of the scene, wherein the scene descriptor data indicates the change of the state of the scene over time;

[0033] Receive a request to render a portion of the scene, wherein the request may include a timestamp;

[0034] Based on the scene descriptor data and the request, render a portion of the scene in the state corresponding to the time stamp to obtain pre-rendered scene data, which includes a pre-rendered version of a portion of the scene.

[0035] The pre-rendered scene data is provided to the rendering device.

[0036] In another aspect of the invention, a rendering system is provided. The rendering system may include:

[0037] Network interface with the network;

[0038] A data storage device including scene descriptor data, wherein the scene descriptor data can identify a set of 3D graphical objects representing at least a portion of the scene, and wherein the scene descriptor data can indicate the change of the state of the scene over time;

[0039] The processor subsystem can be configured to:

[0040] - Receive a request to render a portion of the scene via this network interface, wherein the request may include a timestamp;

[0041] - Based on the scene descriptor data and the request, render a portion of the scene in the state corresponding to the time stamp to obtain pre-rendered scene data, which includes a pre-rendered version of a portion of the scene;

[0042] - The pre-rendered scene data is provided to the rendering device via the network interface.

[0043] In another aspect of the invention, a transient or non-transitory computer-readable medium is provided, the computer-readable medium including data representing a computer program including instructions for causing a processor system to perform any given computer-implemented method described in this specification.

[0044] The above measures involve using scene descriptor data to characterize the scene to be rendered. Specifically, scene descriptor data can identify 3D graphical objects that are part of the scene, for example, by listing object identifiers (on which 3D graphical data of the 3D graphical objects can be retrieved) or by including the 3D graphical data itself that defines the 3D graphical objects. In other words, scene descriptor data can include (external) references to the 3D graphical data of objects, or can contain such 3D graphical data in an embedded form. Such 3D graphical data can, for example, take the form of mesh data defining one or more polygonal meshes of objects. However, it will be understood that 3D graphical objects can also be defined in other ways (e.g., using voxels or point clouds). Typically, such 3D graphical objects can be combined with 2D graphics (e.g., with images or videos such as textures, backgrounds, etc.) for example, during or after rendering.

[0045] Scene descriptor data can also characterize various other aspects of the scene, such as the relative positions and orientations of objects, and the appearance of objects. For example, if an object is animated, the scene descriptor data can define the object's animation in the form of keyframe animation data, or by using one or more transformations.

[0046] Scene descriptor data can further define how the state of a scene changes over time. Therefore, scene descriptor data can define how a scene, or parts thereof, changes over time. Thus, the dynamics of a scene can be defined at least in part by the scene descriptor data. Such dynamics can involve changes within the scene, which can be 'scripted' or predefined in any other way through the scene descriptor data. It will be understood that a scene can additionally or alternatively include non-scripted changes, such as those resulting from user interactions with objects in the scene or from changes based on external events. In some examples, such non-scripted changes to a scene can also be represented in the scene descriptor data, for example, through intermediate updates or revisions to the scene descriptor data.

[0047] Scene descriptor data can be used for rendering devices and remote rendering systems. A rendering device can refer to a 'local' rendering device, as it can be used by an end user and / or included in or connected to a display showing the rendered scene. Specifically, a rendering device can display the rendered scene as an enhancement of a video object or external environment. Here, the term 'video object' can refer to a virtual object representing video, which can simply be a 2D overlay surface, etc., but when the video is displayed in a 3D environment, the virtual object can also be any other type of object, for example, the interior of a sphere shown in omnidirectional video. Furthermore, the term 'external environment' can refer to a part of physical reality, and specifically to the environment surrounding the viewer and / or the rendering device. To enhance such an external environment, the rendering device can include means that overlay the rendered scene onto the external environment in the viewer's eye, for example, using a semi-transparent display or any other type of display suitable for AR or MR rendering.

[0048] Rendering devices can determine which part of a scene can be rendered locally based on scene descriptor data. Such local rendering can involve the rendering device rendering a portion of the scene in real-time or near real-time using known rendering techniques such as rasterization or ray tracing (e.g., using its CPU(s) and / or(s) GPU(s)) in the case of polygon-mesh-based 3D graphics objects. The portion of the scene to be rendered locally (this portion can also be referred to as the 'first' part of the scene) can be identified, for example, by referring to object identifiers listed in the scene descriptor data to identify the objects(s) to be rendered locally. Additionally, the rendering device can determine a portion of the scene to be rendered remotely, which can also be referred to as the 'second' part. The decision to remotely render a portion of the scene, and which portion to remotely render, can be made by the rendering device based on various information such as the complexity of the 3D graphics objects and the computational load of the rendering device.

[0049] After determining a portion of the scene to be remotely rendered, a request can be made to the remote rendering system to render that portion. This may involve sending a request to the remote rendering system over a network. The remote rendering system can also access scene descriptor data. For example, the remote rendering system may represent a 'server,' from which the rendering device acting as a client may have retrieved scene descriptor data. Another example is that both the rendering device and the remote rendering system may have retrieved scene descriptor data from another system or server. In particular, both the rendering device and the remote rendering system can access the scene descriptor data at the start of a rendering session (during which the scene is rendered by the rendering device). Therefore, it may be sufficient for the rendering device to reference a portion of the scene descriptor data in its request for a portion of the remotely rendered scene. In other words, when requesting remote rendering, it may not be necessary for the rendering device to upload mesh data, etc., of 3D graphics objects to the remote rendering system. Instead, for example, the request may reference object identifiers or object locations listed in the scene descriptor data to identify the portion of the scene to be remotely rendered.

[0050] In response, a remote rendering system can use known rendering techniques such as rasterization or ray tracing and its (multiple) CPUs and / or (multiple) GPUs to render portions of the scene to obtain rendered versions of those portions. Such rendered versions can take various forms, such as images or videos, represented in 2D, volumetric 3D, stereoscopic 3D, as point clouds, etc., and can be transmitted to the rendering device over a network in encoded and optionally compressed form. In some examples, the rendered version can also take another form besides images or videos, such as intermediate rendered results representing the output of one or more steps in the rendering pipeline.

[0051] The transmitted rendered scene data can also be referred to as 'pre-rendered' scene data because, from the rendering device's perspective, this rendered scene data can represent a pre-rendered version of a portion of the scene, meaning the rendering device itself may not need to perform (multiple) corresponding rendering steps. However, this does not preclude the rendering device from performing further rendering steps. Typically, when rendering a portion of the scene requires one or more rendering steps, remote rendering can involve at least one of these rendering steps being performed remotely by a remote rendering system.

[0052] After receiving pre-rendered scene data, the rendering device can render the scene as an enhancement of video objects or the external environment using both a locally rendered version of the first part of the scene and a remotely rendered version of the second part. The two rendered versions can be combined in various ways, such as through blending or compositing. In some examples, the rendering device can use a rendering pipeline to render the scene, where the pre-rendered scene data can bypass at least the initial part of the rendering pipeline to be processed by later parts of the pipeline. In another example, the pre-rendered scene data can completely bypass the rendering pipeline so that it can be later combined with locally rendered scene data.

[0053] As indicated above, the scene can be a dynamic scene because the scene descriptor data can at least partially define how the scene changes over time. To ensure, or at least achieve, that the remote rendering system renders a second part of the scene in a time state that matches the time state of the scene rendered by the rendering device, the remote rendering system can be requested to render the second part of the scene at a suitable time instance. Here, the term 'time state' can refer to the state of the scene at a specific point in time. For this purpose, a timestamp can be provided to the remote rendering system. For example, the timestamp can be a timestamp or any other direct or indirect representation of the time instance to be rendered. The remote rendering system can identify the state of the scene at the indicated time instance by accessing the scene descriptor data and can render the second part of the scene accordingly. For example, if a particular 3D graphics object is positioned, oriented, and / or takes a particular pose at the requested time instance, the remote rendering system can render the 3D graphics object accordingly.

[0054] The above measures can be based on the insight that, for dynamic scenes, the hybrid rendering between the rendering device and the remote rendering system should be synchronized. Various ways of doing this can exist. For example, as in [2], the rendering device can maintain the state of the scene and, for example, signal changes in state to the remote rendering system using interaction. However, as explained in the background section, this may not be sufficient for dynamic scenes. According to the above measures, if a request is made to remotely render a portion of the scene, the temporal state of the scene to be rendered can be indicated. This indication of the temporal state, for example, in the form of a timestamp, can be sufficient, since scene descriptor data indicating changes in the scene over time may have been provided to the remote rendering system, for example, before the rendering session begins. This requires relatively little information to be provided to the remote rendering system, reducing bandwidth requirements and / or transmission time. The above measures can also be more robust to interruptions in network connectivity. For example, if a request for remote rendering is lost due to such an interruption, the next request can be processed normally again, whereas in the example of [2], the loss of the request could cause the state of the scene to become out of sync between the remote rendering system and the rendering device, since the request might contain updates to objects. Another advantage is that, for example, the time instance of the second part of the scene to be rendered can be appropriately selected in the future, so that the pre-rendered scene data received by the rendering device can be seamlessly combined with the local rendering output.

[0055] The following embodiments may represent embodiments of a computer-implemented method for rendering a scene using a rendering device, but may also represent embodiments of a rendering device in which its processor subsystem is configured to execute corresponding method steps(s). Unless otherwise excluded for technical reasons, these embodiments may also indicate computer-implemented methods for rendering portions of a scene using a rendering device, as well as corresponding embodiments of a remote rendering system and its processor subsystem.

[0056] In embodiments, scene descriptor data can describe the state of the scene at multiple time instances on the scene timeline to indicate changes in the scene's state over time, and wherein time stamps can indicate time instances on the scene timeline. Therefore, changes in the scene can be linked to the scene timeline, for example, by defining that a particular change occurs at one point on the scene timeline (e.g., at a first time instance) and another change may occur at another point on the scene timeline (e.g., at a second time instance). The scene timeline can be defined, for example, by the scene descriptor data itself, or can otherwise be known jointly by the rendering device and the remote rendering system, thus providing a common reference for remote rendering requests. Therefore, requests from the rendering device relating to a first time state of the scene can be prevented from being incorrectly translated to a second time state of the scene by the remote rendering system due to a lack of common reference. The scene timeline itself can be expressed in any suitable form, including but not limited to timelines expressed in milliseconds, seconds, minutes, etc., or relative to any other reference clock time instance. In particular, the scene timeline can be defined relative to real-time rendering of the scene, since it is known how long the time intervals on the scene timeline should take when the scene is rendered in real time.

[0057] In this embodiment, the 3D graphics object to be rendered may be an adjunct or supplement to the video content. In such a case, the scene's timeline may be at least partially aligned with or referenced in time to the video content's timeline. This can be achieved, for example, by describing scene updates by referencing video decoding or playback timestamps or by controlling video playback based on scene updates. Requests and responses to pre-rendered scene data can be synchronized using video timestamps.

[0058] In embodiments, scene descriptor data may indicate at least one of the following: the presence, location, orientation, and appearance of a corresponding object in the scene as it changes over time. A scene can change in various ways, including but not limited to the appearance or disappearance of objects from the scene, changes in the location of individual objects or portions of such objects, changes in the orientation of individual objects or portions of such objects, and changes in how objects behave in the scene, such as their pose or other external attributes. Typically, such changes are defined by defining changes in object or scene properties or by redefining object or scene properties for a specific time instance. Scene descriptor data may define such changes directly (e.g., by defining changes in properties or by defining new values ​​for properties) or indirectly. An example of the latter is scene descriptor data that contains references to different versions of the scene for corresponding different time instances, wherein such versions of the scene are defined elsewhere, for example, by defining scene data for each version of the scene. Scene descriptor data may be linked to such scene data.

[0059] In an embodiment, the method may further include determining a timestamp to indicate a second time instance that is temporally earlier than a first time instance of the scene currently being rendered by the rendering device. By being able to indicate a time instance of a second portion of the scene to be remotely rendered by a remote rendering system, the rendering device can request rendering of a second portion of the scene for a time instance that is temporally earlier than the time instance of the scene currently being rendered by the rendering device. This allows the rendering device to anticipate that it may receive the requested pre-rendered scene data with only a certain delay, for example, due to network or processing latency. This reduces or avoids the need for techniques such as post-projection, which might otherwise have to be used to compensate for such latency by transforming the scene rendered for a past time instance to the current time instance, and these techniques can be computationally complex and introduce artifacts.

[0060] In an embodiment, the method may further include determining a timestamp based on at least one of the following:

[0061] - Network latency between the rendering device and the remote rendering system;

[0062] - The latency or associated scheduling constraints of rendering the second part of the scene by the remote rendering system.

[0063] A time instance for rendering the second part of the scene by a remote rendering system can be selected to specifically account for various latency issues in the network and / or processing. For example, such latency can be measured or estimated, for instance, by a rendering device, and timestamps can be generated to compensate for said latency by requesting the rendering of the second part of the scene(s) for a given time instance (a time earlier than the current time instance of the scene being rendered, corresponding to such latency(s)). In some embodiments, timestamps can be generated to account for the measured and / or estimated latency(s), while additional safety margins are added, for example, to account for various smaller or unknown latency issues. By determining timestamps based on such estimated and / or measured latency(s), rendered scene data can be prevented from arriving too late (which may necessitate the use of techniques such as post-reprojection discussed above) or significantly too early (which may require longer buffering times), and this may reduce the ability to allow for sudden and dynamic changes in the scene.

[0064] It should be noted that, for example, as a result of a remote rendering system predicting that a rendering client needs remote rendering assistance, any request to render a part of the scene at a certain time instance (e.g., a current time instance that may precede the current rendering scene currently being rendered by the rendering device) can be generated internally by the remote rendering system. Such a prediction can be based, for example, on historical data of past requests from the rendering device, or on (multiple) current requests from the rendering device.

[0065] In an embodiment, scene descriptor data may include object identifiers of corresponding graphics objects in the set of 3D graphics objects, wherein:

[0066] - Requesting a remote rendering system may include providing the remote rendering system with corresponding object identifiers for one or more 3D graphics objects representing a second part of the scene; and / or

[0067] Receiving pre-rendered scene data may include receiving metadata of the corresponding object identifiers of one or more 3D graphics objects pre-rendered in the pre-rendered scene data.

[0068] By providing the object identifiers of the corresponding graphical objects in the set of 3D graphical objects as part of the scene descriptor data, the rendering device can request the rendering of the (second) part of the scene by simply providing the corresponding object identifiers to the remote rendering system. This requires providing relatively little information to the remote rendering system, reducing bandwidth requirements and / or transmission time. Alternatively, the remote rendering system can provide metadata along with the pre-rendered scene data, which may include the corresponding object identifiers of one or more 3D graphical objects pre-rendered in the pre-rendered scene data. This informs the rendering device which rendering objects are included in the pre-rendered scene data, and can be advantageous if, for example, the pre-rendered scene data contains fewer rendered objects than requested. For example, the remote rendering system may be unable to render all requested objects, for example, due to computational limitations. The rendering device can be informed via metadata, and if object identifiers are missing from the metadata, the rendering device can override this omission by rendering the corresponding objects locally. This allows the rendering device to adapt to unforeseen changes or deficiencies in the remote rendering of the remote rendering system. This can be particularly advantageous if the pre-rendered scene data contains, or in principle can contain, multiple rendered objects. In such a situation, one can expect to know which objects are rendered and which are not rendered in the pre-rendered scene data, and which objects may not be easily determined from the pre-rendered scene data itself.

[0069] In an embodiment, metadata may be repeatedly received by the rendering device, and the method may further include stopping or starting local rendering of 3D graphics objects based on object identifiers of 3D graphics objects that appear in or disappear from the metadata. The remote rendering system may provide metadata on a periodic, semi-periodic, or otherwise repetitive basis, such as a metadata stream accompanying a video stream of pre-rendered scene data. By monitoring the metadata to detect object identifiers appearing or disappearing from it, the rendering device can detect when and which object is being rendered in the pre-rendered scene data, and can accordingly adapt its local rendering, for example, by starting local rendering of the object if it disappears from the pre-rendered scene data, or by stopping such local rendering.

[0070] In embodiments, scene rendering can be adapted to the content of a video object or external environment, and the method may further include determining content data that at least partially characterizes the content and providing the content data to a remote rendering system. Scene rendering can be adapted to the content of the video object or external environment to be enhanced. Such adaptation can, for example, improve the realism of the enhancement to convey the impression that objects in the scene belong to the content of the video object or external environment. A specific example could be that the scene can be rendered as an enhancement of an outdoor environment at sunset. The scene can be rendered to simulate this sunset scene, for example, by adding directional light sources at appropriate locations in the scene and selecting the color temperature of the light sources to match the sunset. The nature of the content of the video object or external environment may be available to the rendering device, for example, based on (sensor) measurements, but may not typically be available to a remote rendering system. However, to enable a remote rendering system to adapt its rendering of a portion of the scene to such properties, the rendering device can determine content data that at least partially characterizes the content of the video object or external environment, for example, in terms of lighting conditions, and can provide the content data to the remote rendering system so that the remote rendering of a portion of the scene can be adapted to such properties.

[0071] In this embodiment, the content data may include at least one of the following:

[0072] - Illumination data representing the lighting in this content;

[0073] - Depth data representing the depth of the content; and

[0074] - Provides image or video data that gives a visual representation of the content.

[0075] Therefore, a rendering device can generate lighting data, such as orientation, intensity, and color temperature, characterizing the lighting in a video object or external environment, and provide this data to a remote rendering system so that the system can adjust its rendering accordingly. Another example is that the rendering device can generate data, for example, representing the depth of the content in the form of a depth map, which can be generated from the content through depth estimation. For example, such depth estimation can be performed using a stereo camera or time-of-flight sensor connected to or integrated into the rendering device. This allows the remote rendering system to adapt its rendering to the depth of the content to be enhanced, for example, by adjusting the projection matrix of the rendering. Yet another example is that the rendering device can provide a visual representation of the content to a remote rendering system so that the remote rendering system itself can detect relevant properties of the content. For example, the rendering device can provide a transcoded version of a video object to the remote rendering system, for example, in a reduced-quality version, which may be sufficient to determine lighting conditions, depth, etc. Yet another example is that the rendering device can capture the external environment using a camera and stream the resulting video, for example, in reduced quality, to the remote rendering system. Another example is that a rendering device can use a camera and / or other sensors to capture the external environment, and can perform image recognition locally to extract features from the captured content, and can transmit data characterizing the extracted features as content data to a remote rendering system.

[0076] In an embodiment, determining a first portion of the scene to be rendered locally and a second portion of the scene to be rendered remotely may include determining whether to render the 3D graphics object locally or remotely based on at least one of the following:

[0077] - The latency, bandwidth, or reliability of the network between the rendering device and the remote rendering system;

[0078] - The latency, bandwidth, or reliability of the network between the rendering device and the content server, which is configured to host the 3D graphics data of the set of 3D graphics objects that define the scene;

[0079] - The computational load of the rendering device;

[0080] - The computational load of the remote rendering system;

[0081] - The battery level of the rendering device;

[0082] -The complexity of the 3D graphics object; and

[0083] - The scene distance between the viewpoint of the rendering device when rendering the scene and the 3D graphics object in the scene.

[0084] Various types of information can be considered to determine which part of the scene to render remotely and which part to render locally. It should be noted that the decision of where and which part of the scene to render can be made on a periodic, semi-periodic, or repetitive basis (e.g., every second, every ten seconds, every minute, etc.) or upon detecting an event (which enables the rendering device and / or remote rendering system to react to sudden changes (e.g., sudden increases or decreases in computational load, increased packet loss, etc.)). This ensures, or at least benefits, that the rendering device can render the scene even in the event of such sudden changes.

[0085] In an embodiment, scene descriptor data may include a graph containing branches indicating hierarchical relationships between 3D graphics objects, wherein determining a first part of the scene to be rendered locally and a second part of the scene to be rendered remotely may include determining whether each branch's 3D graphics objects are to be rendered locally or remotely. The graph may be a suitable data representation of relationships between objects and / or parts of objects in the scene. For example, objects or object parts may demonstrate connectivity in the scene, such as spatial connectivity, by actually being connected to each other or by being in proximity to each other or by interaction between objects. Such connectivity can be well represented as edges in the graph. When rendering the scene, for example, for rendering efficiency or quality, it may be desirable to render such connected objects or object parts by a single entity. Therefore, when determining where to render which part of the scene, a decision can be made on a branch-by-branch basis, such connected objects or object parts are rendered locally by a rendering device or remotely by a remote rendering system. This avoids rendering inefficiencies or reduced rendering quality. In this respect, it should be noted that scene descriptor data may also indicate hierarchical relationships between 3D graphics objects on one side and other types of assets such as textures, audio, etc. on the other side.

[0086] The following embodiments may represent embodiments of a computer-implemented method for rendering a portion of a scene for a rendering device, but may also represent embodiments of a remote rendering system, wherein its processor subsystem is configured to execute corresponding method steps(s). Unless otherwise excluded for technical reasons, these embodiments may also indicate computer-implemented methods for rendering a scene using a rendering device, and corresponding embodiments of the rendering device and its processor subsystem.

[0087] In an embodiment, the method may further include generating metadata containing corresponding object identifiers for one or more 3D graphics objects pre-rendered in the pre-rendered scene data, and providing this metadata along with the pre-rendered scene data to the rendering device. As discussed elsewhere, such metadata enables the rendering device to detect object identifiers appearing or disappearing from the metadata. Therefore, the rendering device can detect when and which object is being rendered in the pre-rendered scene data, and can accordingly adapt its local rendering, for example, by starting local rendering of the object when it disappears from the pre-rendered scene data, or by stopping such local rendering.

[0088] In an embodiment, the method may further include adjusting a portion of the scene to be rendered by omitting 3D graphics objects from the rendering based on one of the following:

[0089] - The computational load of the remote rendering system;

[0090] - The latency, bandwidth, or reliability of the network between the rendering device and the remote rendering system.

[0091] The aforementioned measures enable the remote rendering system to determine whether and to what extent it should assist in rendering the scene, allowing it to take into account information unavailable to the rendering device (such as its own computational load). While the rendering device can request the remote rendering system to render a portion of the scene, for example, if the remote rendering system's computational load is too high or if the network connection to the rendering device is too poor to reliably transmit the pre-rendered scene data for those objects, the remote rendering system can omit one or more objects from the rendering. Therefore, the remote rendering system can partially or, in some cases, completely reject a request for remote rendering. In the case of a complete rejection (where the remote rendering system decides not to render any of the requested objects), the remote rendering system can respond to the request with a rejection. In the case of a partial rejection (where the remote rendering system decides to render only a subset of the requested objects), the remote rendering system can respond to the request using a response indicating that only a subset of the requested objects will be rendered and / or by providing metadata along with pre-rendered scene data identifying objects that have already been rendered, thereby implicitly indicating to the rendering device which objects have not yet been rendered.

[0092] Those skilled in the art will understand that two or more of the embodiments, implementations and / or aspects of the invention mentioned above can be combined in any manner that they deem useful.

[0093] Modifications and changes to any of the systems or devices (e.g., remote rendering systems, rendering devices, etc.), computer-implemented methods, metadata, and / or computer programs (corresponding to the modifications and changes described in another of these systems or devices, computer-implemented methods, metadata, and / or computer programs, and vice versa) may be performed by a person skilled in the art based on this specification. Attached Figure Description

[0094] These and other aspects of the invention will be understood and illustrated with reference to the embodiments described below. In the accompanying drawings,

[0095] Figure 1 A rendering device configured to request remote rendering of a portion of a scene is shown, and a remote rendering system configured to render the scene portion in response to the request and provide pre-rendered scene data to the rendering device is also shown.

[0096] Figure 2 A scene is shown that uses a head-mounted display for rendering to enhance the external environment, in which a first object is rendered locally and a second object is rendered remotely by a remote rendering system.

[0097] Figure 3 This demonstrates a request to remotely render a portion of the scene for a time instance that precedes the current time instance in time.

[0098] Figure 4 The steps of a computer-implemented method for blended rendering of a scene are shown, including steps performed at or by a local rendering device.

[0099] Figure 5 The steps of a computer-implemented method for blended rendering of a scene are shown, as well as the steps performed at or by a remote rendering system.

[0100] Figure 6 A non-transitory computer-readable medium including data is shown;

[0101] Figure 7 An exemplary data processing system is shown.

[0102] It should be noted that items with the same reference numerals in different figures have the same structural features and the same function, or the same signal. Since the function and / or structure of such items have already been explained, it is unnecessary to repeat the explanation in the specific implementation.

[0103] List of reference numerals

[0104] The following list of reference numerals and abbreviations is provided to facilitate the interpretation of the drawings and should not be construed as limiting the claims.

[0105] 10 Network

[0106] 20 Scene Descriptor Data

[0107] 40. Requested data representation

[0108] 60 Pre-rendered scene data

[0109] 100 rendering devices

[0110] 120 Network Interface

[0111] 140 Processor Subsystem

[0112] 160 Data storage devices

[0113] 180 Display Interface

[0114] 182 Display Data

[0115] 184 sensor data

[0116] 190 Head-mounted Display

[0117] 200 Remote Rendering System

[0118] 220 Network Interface

[0119] 240 Processor Subsystem

[0120] 260 Data storage devices

[0121] 300 users

[0122] 320 Head-Mounted Display

[0123] 340 External Environment

[0124] 342 Field of View

[0125] The part of the scene that 360 needs to render locally

[0126] 380 parts of the scene to be rendered remotely

[0127] 400 Request to render scene portion at location n+3

[0128] 410 renders the scene at location n+3.

[0129] 420 transmits rendered scene data for n+3.

[0130] 430 renders the scene at location n+3.

[0131] 440 shows the scene at position n+3.

[0132] 510 Obtain scene descriptor data

[0133] 520 determines the scene portion used for remote rendering.

[0134] 530 Request remote rendering and provide time stamps

[0135] 540 Obtain pre-rendered scene data

[0136] 550 Rendering Scene

[0137] 610 Obtain scene descriptor data

[0138] 620 Receive rendering request

[0139] 630 Rendering Scene Section

[0140] 640 provides pre-rendered scene data.

[0141] 700 Non-transitory computer-readable media

[0142] 710 Data

[0143] 1000 Exemplary Data Processing Systems

[0144] 1002 processor

[0145] 1004 Memory Element

[0146] 1006 System Bus

[0147] 1008 Local Memory

[0148] 1010 High-capacity storage device

[0149] 1012 Input Devices

[0150] 1014 Output Device

[0151] 1016 Network Adapter

[0152] 1018 Application Detailed Implementation

[0153] The following embodiments may involve using a rendering device to render a scene, which can be configured to display video objects or allow the external environment to be visually conveyed to the viewer and render the scene as a video object or an enhancement of the external environment.

[0154] The scene to be rendered can be defined in various ways. For example, the scene can include 3D graphics objects, which can be defined by geometric primitives such as polygons. As is known per se, a polygon mesh can be defined, for example, by vertices, edges forming connections between vertices, faces, surfaces, materials, UV coordinates, etc. When 3D graphics objects are defined as geometric primitives, the rendering of the scene can, for example, involve rasterization or ray tracing, both of which are known per se. The result of such rendering can be a bitmap or any other form of raster graphics, which can be written to a frame buffer for display on a monitor. It should be noted that such 3D graphics objects can also include, for example, video in the form of textures applied to the 3D graphics objects. In a particular example, the 3D graphics object to be rendered can be a sphere, where the interior of the sphere shows omnidirectional video. Therefore, the rendering of the 3D graphics object can include rendering video. Alternatively, or as an alternative to using geometric primitives, 3D graphics objects can also be defined in other ways (e.g., as point clouds or by voxels), both of which can be rendered to obtain a bitmap to be displayed.

[0155] Rendering can involve various steps. For example, in the case of rasterization-based rendering, individual rendering steps may include vertex processing, tessellation, primitive processing, kernel rasterization, fragment processing, and pixel processing. When it comes to rendering a scene, such a reference can refer to all rendering steps, but it can also refer to a subset of such rendering steps, such as one or more of them. Typically, rendering can be performed using one or more CPUs and / or one or more GPUs.

[0156] The following embodiments utilize scene descriptor data to render a scene. Such scene descriptor data can, for example, represent a description of the scene by identifying 3D graphical objects in the scene. For example, scene descriptor data may include object identifiers for the respective objects, while also including or relating to graphical data defining each object. Typically, scene descriptor data can model the spatial geometry of the scene with reference to spatial relationships between objects (e.g., their relative positions and orientations). As is known in itself, for example, in section 4.6.6 of reference [1], the spatial geometry of the scene can be modeled in various ways, such as as scene graphs. For example, scene graphs can be directed acyclic graphs, such as a naive tree structure representing the geometry of the scene based on an object-based hierarchy. For example, leaf nodes of the graph can represent geometric primitives such as polygons, for example, by including identifiers of geometric primitives. Spatial transformations can be represented as nodes of the graph and are represented by transformation matrices. Other scene graph nodes can represent light sources, particle systems, virtual cameras, etc.

[0157] Figure 1An example rendering device 100 is shown, which may include a network interface 120, a processor subsystem 140, a data storage device 160, and a display interface 180, each of which is described in more detail elsewhere in this specification. The rendering device 100 can access scene descriptor data 20 describing the scene to be rendered, and can determine, based on the scene descriptor data 20, a first portion of the scene to be rendered locally by the rendering device and a second portion of the scene to be rendered remotely. For example, the rendering device 100 may determine to render one or more objects locally and render the remaining objects remotely, or vice versa, and may request a remote rendering system 200 to render the second portion of the scene. For this purpose, the rendering device 100 may send a request 40 to the remote rendering system 200, for example, via the network interface 120 of the rendering device 100 and via network 10 connecting the rendering device 100 to the remote rendering system 200, for receipt by the network interface 220 of the remote rendering system 200. The remote rendering system 200 can also access the scene descriptor data 20, for example, by acting as a host for the scene descriptor data 20 retrieved by the rendering device 100, or by retrieving the scene descriptor data 20 from elsewhere. Therefore, the remote rendering system 200 can render a second portion of the scene to obtain pre-rendered scene data 60 (including a pre-rendered version of the second portion of the scene), and send the pre-rendered scene data 60 to the rendering device 100 via the network 10. The rendering device 100 can then render the scene as an enhancement of a video object or external environment, i.e., by locally rendering a first portion of the scene and by including a pre-rendered version of the second portion of the scene, for example, by mixing or otherwise combining the pre-rendered version of the second portion of the scene with the locally rendered first portion.

[0158] Such hybrid rendering can involve rendering on a frame-by-frame basis, meaning that rendering device 100 can render a first portion of a scene for a given frame, receive a second portion of the scene as pre-rendered scene data for that given frame, combine the two rendered scene portions, and then display the resulting combined frame. However, it will be understood that remote rendering can, but does not need to, be requested frame by frame. For example, rendering device 100 can request remote rendering to begin at a specified time instance and continue for a specified duration or an unspecified duration until rendering device 100 requests to stop remote rendering, and remote rendering system 200 can begin rendering frames backward from the specified time instance. The pre-rendered scene data can then be streamed to rendering device 100, for example, as a video stream. In a particular example, the video stream can be encoded using adaptive streaming techniques such as, but not limited to, MPEGDASH (Dynamic Adaptive Streaming over HTTP).

[0159] exist Figure 1In the example of the embodiment, rendering device 100 may represent an AR client and may perform the following steps or a subset thereof to achieve hybrid rendering. Here and elsewhere, rendering device may be simply referred to as 'client', and remote rendering system may be simply referred to as 'rendering server', while rendering context may be AR, but may also be generally applied to XR or VR.

[0160] 1. The client may obtain the AR scene descriptor file of the scene to be rendered, for example, from a rendering server or from a content server hosting the scene's assets. The AR scene descriptor file may be an example of scene descriptor data as described elsewhere in this specification.

[0161] 2. From the AR scene descriptor file, the client can identify and download 3D assets, for example, from a rendering server or a content server. In some examples, the client can download all 3D assets of the scene, while in other examples, the client can only download a portion of all 3D assets, such as only those 3D assets visible from the user's viewpoint in the scene.

[0162] 3. Based on criteria such as the user's location in the scene, the available bandwidth between the server and the client, and the available resources at the server and / or client, the client can determine which(s) the rendering server should render locally and which(s) the rendering server should render remotely.

[0163] 4. The client may request the rendering server to render selected objects(s), and provide pre-rendered scene data to the client, for example, as a video stream. In or accompanying the request, the client may include information that may be related to parts of the scene being rendered by the rendering server, such as information defining the user's viewpoint in the scene, for example, in the form of viewing position and / or orientation. Additional information may include content data characterizing lighting conditions in the content to be enhanced, user preferences, transcoded versions of the video objects to be enhanced, or video recordings from outward-facing and / or inward-facing cameras showing the external environment to be enhanced.

[0164] 5. The client can receive and decode the video stream to obtain a decoded version of the pre-rendered scene data containing pre-rendered content.

[0165] a. If the pre-rendered content still needs to undergo further rendering steps before being displayed, such pre-rendered content can be fed into the client's rendering engine, but multiple rendering steps, such as one or more initial rendering steps, can be bypassed.

[0166] b. If the pre-rendered content represents (multiple) fully pre-rendered objects, the image or video data of that pre-rendered content can be copied to a video buffer for merging after the rendering engine generates the so-called viewport. Such pre-rendered content can include, but is not limited to:

[0167] i. Traditional image or video content (e.g., single-view, stereoscopic, multi-view, volumetric, and / or omnidirectional).

[0168] ii. Render-ready content that can be directly copied to a buffer (e.g., a vertex or fragment buffer for OpenGL). Such pre-rendered content can be, for example, in the form of textures, sprites, or any other low-level graphics elements.

[0169] iii. Possesses any one of the above abilities to process metadata (transformation matrix, transparency layer, clipping / scaling properties, etc.).

[0170] 6. The client's rendering engine can create a partially rendered viewport by locally rendering a selected number of objects.

[0171] 7. If some objects are available in a pre-rendered format, the client's rendering engine can merge a partially rendered viewport with a video buffer containing the pre-rendered objects(s) to generate the final rendered viewport.

[0172] 8. The client can start again from step 3 until the AR presentation ends.

[0173] exist Figure 1 In another example of the embodiment, rendering device 100 may represent an XR client, such as a pair of tethered AR glasses that can be configured to render spatially aligned 3D and 2D assets in an external environment (e.g., in the viewer's physical surroundings). Remote rendering system 200 may act as a rendering server for the XR client and may include the content of the scene to be rendered, including descriptor files, 3D / 2D assets representing 3D / 2D graphics objects, etc., and as previously described, may have the ability to render portions of the scene, such as omnidirectional frames, encode / transcode rendered portions of the scene and any other generated assets, encapsulate and distribute them. Such an XR client and rendering server may conform to the hybrid rendering category of section Fout! Verwijzingsbron niet gevonden. in reference [1]. It should also be noted that the following references to XR clients and rendering servers, as well as embodiments of XR clients and rendering servers, can also be applied to any other type of rendering device and any other type of remote rendering system as described in this specification.

[0174] During operation, the XR client can retrieve one or more descriptor files containing information about the scene and available and / or required assets. The assets themselves may be stored at / by a rendering server, or at / by another server, which is also referred to elsewhere as a 'content server'. The descriptor files may include or represent scene descriptor data as described elsewhere. Based on the scene descriptor data, the XR client can download those assets required to render the scene. In some embodiments, the XR client can download all assets to enable offline rendering of the scene, for example, even if a network connection is interrupted. Alternatively, the assets and descriptor files may be stored on the XR client and uploaded to the rendering server, for example, before the rendering session begins.

[0175] After parsing, for example, multiple descriptors contained in scene descriptor data, the XR client can select which of the required assets to render, based on criteria such as visibility. The XR client can then decide which of the selected assets to render locally and which to render remotely. The criteria for local / remote selection can be multi-dimensional, for example, based on geometric complexity, network connectivity, semantic importance, spatial location, etc. After making this decision, the XR client can send a request to remotely render the selected assets. Such a request can consist of an retrieval request to extract the remotely rendered assets. Simultaneously, earlier, or later, the XR client can begin local rendering of the selected assets. The XR client's request can contain information about the rendering context, such as indicating virtual camera position, head pose, etc., and may include environmental information captured by sensors located on the client (such as an ambient light sensor capturing light data, a depth sensor capturing depth data, or an outward-facing camera recording images or video). The rendering server can then render the requested portion of the scene based on the received request and associated information. After rendering, the pre-rendered scene data, along with accompanying information such as synchronization information, transformation matrices, and codec properties, can be compressed and sent to the XR client. This accompanying information can be transmitted to the XR client in the form of metadata. When the XR client has received the pre-rendered scene data and when locally rendered scene data is available, these two types of data can be combined, for example, by overlaying or stitching. For this purpose, both types of data can be temporally and spatially aligned, which could potentially be the product of processing based on the accompanying information received from the rendering server.

[0176] The division between assets to be rendered locally and assets to be rendered remotely can be adjusted during the rendering session (e.g., at any time). Therefore, it is desirable to keep scene descriptor data synchronized between the XR client and the rendering server. For this purpose, when a scene update occurs (which can be initiated by the XR client, rendering server, content server, or remote service, for example, based on user input or an external event), the scene descriptor data at both entities can be updated, for example, by transferring the scene description update ('differential') and optionally the new assets associated with the scene description update from one entity to another. Such an update mechanism can also be used if communication between the rendering server and the XR client is interrupted but subsequently re-established. Alternatively, in cases where a scene description update cannot be applied, for example due to compromised file integrity or expiration, the most recent version of the scene description can be requested.

[0177] Figure 2 An example of a scene rendered using a head-mounted display to enhance the external environment is shown. Here, a head-mounted display 320 to be worn by a user 300 is shown, wherein the head-mounted display 320 allows the user 300 to view the external environment 340, for example, through a transparent or semi-transparent portion of the head-mounted display 320. The rendering device itself is not shown. Figure 2 As shown, but in some embodiments can be integrated into the head-mounted display 320, which may then also be referred to as a head-mounted device, while in other embodiments the rendering device may be separate from the head-mounted display 320, but connected to it, for example, using wired or wireless communication. The rendering device can render a scene to the user to enhance the user's view of the external environment 340. For example, such enhancement may occur within a field of view 342 corresponding to the field of view provided by the transparent or translucent portions of the head-mounted display 320, or within its sub-portions. Figure 2 As shown, the first part 360 of the scene can be rendered locally by a rendering device, while the second part 380 of the scene can be rendered remotely by a rendering system. Figure 2 (Not shown in the image) Remote rendering. For example, a rendering device can decide to locally render a 3D graphics object that looks closer to the user, for example, because the user can interact with the 3D graphics object, and that interaction can benefit from lower latency, while another 3D graphics object that looks farther away from the user can be rendered remotely by a remote rendering system.

[0178] Application Examples

[0179] Typically, there can be various application examples where rendering scenes exist to enhance video objects or the external environment. Although Figure 2This illustrates an AR gaming experience where users can battle virtual monsters superimposed on their actual physical surroundings, but various other types of applications are possible, such as in tourism, commerce, industry, and education. One such example could be a museum with AR-rendered paintings. For this purpose, a physical museum could include empty canvases onto which paintings could be AR-overlaid using a head-mounted display worn by the user. Thus, these paintings could represent 3D graphical objects of the scene, where individual paintings are defined by frames and textures that depict the paintings. Users could physically walk through the museum, and this movement could be tracked by a rendering device to adjust the overlay of the paintings based on the user's position and viewing direction. This allows users to move freely through the museum and view the paintings from different angles. AR content (i.e., the overlaid paintings) could be rendered entirely remotely by a remote rendering system, entirely locally by a rendering client, or in a hybrid manner. For example, distant objects could be rendered by a remote rendering system, while nearby objects could be rendered by a rendering device. The rendering device could determine what is nearby and what is far based on the user's position relative to the objects.

[0180] In a specific example, a user can start at position A1 and move to position A2. At position A1, the user can move away from object 1, making local rendering of object 1 unnecessary since distant objects require less detail. Thus, a remote rendering system can be requested to render object 1 remotely, and a video stream showing a rendered version of object 1 from the user's perspective will be transmitted to the rendering device. When moving closer to object 1, a higher resolution rendering of object 1 can be expected, allowing the rendering device to switch to local rendering.

[0181] Signaling for remote and local rendering

[0182] Hybrid rendering can involve, for example, signaling for clients to request remote rendering and for rendering servers to respond to such requests (e.g., by acknowledging such requests and by sending pre-rendered scene data to the clients). The signaling described below can use any suitable protocol and / or syntax. For example, signaling can be based on WebSocket, or integrated into RTSP / RTCP control messages, or utilize HTTP REST APIs or other protocols. The various signaling protocols described below focus on the information contained in such messages rather than on the signaling protocols themselves.

[0183] By default, the client can control whether rendering is performed locally, remotely, or in a hybrid manner. Exceptions to this default are described elsewhere, such as when the rendering server is unable to perform all requested remote renderings due to insufficient computing resources.

[0184] Switching between local rendering and remote rendering

[0185] The table below illustrates the messages used to enable or disable remote rendering. A first message can be sent from the client to the rendering server to initiate remote rendering. This message can contain various information as defined below. It will be understood that if certain information is stated as 'required' here or elsewhere, this only applies to this particular example of the message, as such information may not be required in other types of messages.

[0186]

[0187] Here, the term 'position' can refer to a location within the scene, and more specifically, to the location of the viewport from which the scene is rendered. This position can be defined in 3D coordinates, such as x, y, z coordinates, but can also be defined in any other way, such as 2D coordinates. Similarly, the term 'orientation' can refer to the orientation of the viewport from which the scene is rendered, and this orientation can be defined in any suitable way, such as the corresponding yaw, pitch, and roll angles.

[0188] Using this information, the rendering server can begin creating the requested video stream for the requested location in the scene. The rendering server can confirm the request to the client using the following message:

[0189]

[0190] When a client wants to stop remote rendering of (multiple) objects, it can send the following message to the rendering server:

[0191]

[0192] After setting up remote rendering, the client can periodically, semi-periodically, or repeatedly send messages to the rendering server to update the client's position in the scene, for example, using the following messages:

[0193]

[0194] Conversely, the rendering server can respond using a message that instructs the client to retrieve the new endpoint of the updated video stream:

[0195]

[0196]

[0197] Modeling Temporal Behavior

[0198] In addition to defining a scene based on the spatial arrangement of objects, scene descriptor data can also define changes in the scene over time. For example, scene descriptor data can describe one or more of the following: the presence, location, orientation, and appearance of a corresponding object at one or more time instances to define changes in the scene over time. Therefore, the temporal state of the scene as it changes over time can be described. In a specific example, such changes can be described at multiple time instances on or for multiple time instances on the scene timeline. Modeling the temporal dynamics of a scene in this way can be advantageous for dynamic scenes. For example, a scene can represent a dynamic 3D environment such as a science museum, where objects can move around and users can interact with them. To define which object is in which location within the scene, a clock and a reference time can be used. The reference time can be, for example, referred to as a presentation timestamp ('pts' or 'PTS') similar to the presentation timestamp when playing media content such as an MPEG DASH video stream. Scene descriptor data can define changes in the scene by referencing such a presentation timestamp. For example, scene descriptor data can define multiple events for different time instances of each or more objects, while also defining the possible different locations of the (multiple) objects for each of these events, as illustrated in the table below. Here, the 'event' tag can be used to define the event's presentation timestamp and define the event itself.

[0199] Therefore, in the following example, an object with object identifier "1" is defined as moving on the x-axis between PTS=0 and PTS=2, and then moving again at PTS=9.

[0200] <object id＝"1">

[0201] <eventpts="0"x="0"y="0"z="0" / >

[0202] <eventpts="1"x="1"y="0"z="0" / >

[0203] <eventpts="2"x="2"y="0"z="0" / >

[0204] <eventpts="9"x="3"y="0"z="0" / >

[0205] < / object>

[0206] Scene descriptor data of this type and similar can enable a client to request a rendering server to render a portion of the scene at a specific time instance. In some embodiments, a client can request the rendering server to render a portion specifically at a future time instance, which refers to a time instance in which the client renders before the current rendering time instance. This accommodates latency in the network between the client and the rendering server, and / or latency in the rendering system, and / or other latency in the 'end-to-end' chain from remote rendering to displaying the remotely rendered content.

[0207] Figure 3This illustrates a request for remotely rendering a portion of a scene for a time instance that precedes the current time instance of the rendered scene. Here, a subset of the steps performed by the rendering device 100 as a client and the remote rendering system 200 as a rendering server are shown. These steps include the rendering device 100 locally rendering the contents of consecutive output frames n, n+1, n+2, n+3, etc., and then displaying the rendered content after rendering. It can be decided that the rendering system 200 will assist in the rendering of the scene. Therefore, while rendering frame n, the rendering device 100 can send a request 400 to the rendering system 200 to render one or more objects at the time instance corresponding to frame n+3. Upon receiving this request, the rendering system 200 can then render 410 this portion of the scene at the specific time instance, thereby obtaining pre-rendered scene data for that time instance. Then, transmission 420 and the rendering device 100 receiving the pre-rendered scene data on time can be performed to display the pre-rendered scene data together with the content locally rendered 430 for frame n+3.

[0208] synchronous

[0209] There are several ways to enable a client to accurately combine remotely rendered objects with locally rendered objects both spatially and temporally. The following discussion considers how to ensure time alignment, which can also be referred to as 'synchronization'. Here, it is assumed that both remote rendering by the rendering server and local rendering by the client produce corresponding video frames to be synchronized. However, it will be understood that rendered objects can also be generated as another data structure besides frames.

[0210] Using a clock: The entity generating the frames (such as a client or rendering server) can correlate the generated frames with an internal clock, for example, by generating a rendering timestamp using the clock and marking the generated frames with this rendering timestamp. However, the two internal clocks may not need to be synchronized. Instead, for each entity, knowing the start time of the clock, defining the increment of the rendering timestamp per frame, and defining the frame rate when the scene starts rendering may be sufficient. Thus, the rendering timestamp generated by the rendering server and transmitted to the client along with the frames can be converted by the client into a rendering timestamp generated by the client. An example could be as follows: A client can start rendering a scene with object A rendered by the rendering server and object B rendered locally. The scene descriptor could describe that at PTS=10, both object A and object B will rotate 90 degrees. To synchronize this, the client might need to know how the frames received by object A are temporally correlated with the locally generated frames of object B, and then the frames received by object A and the locally generated frames of object B can be combined, for example, by overlaying, blending, or otherwise. For this purpose, the frame rate and the increment of the clock per frame can both be defined in the descriptor file. As an example, with a frame rate of 30 and an increment of 1 per frame, when a client receives a frame with PTS = 10, the client can know that the frame is for its own PTS, which is equal to: current_time_wallclock – start_time_wallclock + (1 / 30 * 10).

[0211] Using timeouts: You can define the maximum duration (or response time) for the rendering server to respond to a client's request. This maximum duration or response time can be signaled to the rendering server. If a response from the rendering server containing the remotely rendered frame arrives after this maximum duration or response time, the response can be considered lost / invalid. In such an approach, the request can be related to the rendering of a single frame.

[0212] Using frame numbers: Each request may include a frame number, which can be the frame number expected to be rendered remotely. After receiving the pre-rendered frame, the client can compare the frame numbers to match the remotely rendered frame with the locally rendered frame. For such a method, it is preferable that the frame rate is known at both the rendering server and the client.

[0213] Using an asynchronous API: In this approach, the communication channel between the rendering server and the client can be open, and events can be triggered via the API whenever rendered output from the rendering server is ready. In response, the client can resolve event properties via the API, which may include object identifiers, frame numbers, and / or other information.

[0214] In some embodiments, such as after the client disconnects from the rendering system and when the user can interact with the object, the client can signal the rendering server to notify of the object's changed properties. Such signaling can be performed in a manner similar to the signaling described previously, but may include, for example, the following types of information:

[0215]

[0216] Switch the logic from local rendering to remote rendering and return.

[0217] During a hybrid rendering session, the rendering of one or more objects may switch from local rendering to remote rendering and vice versa. This can happen multiple times (if not many times) during the rendering session. To make such switching as seamless as possible, the client may need to determine when to stop locally rendering objects and instead use an incoming video stream containing the rendered objects. That is, if such local rendering is stopped before remotely rendered objects become available, the user may experience objects disappearing and reappearing, or may perceive the same objects twice, for example, placed on top of each other.

[0218] Therefore, it is expected that it is possible to know when to start or stop rendering objects locally at the client. This can be known by knowing the identifier of the object (to which rendering will switch from local to remote, or vice versa) and a PTS or other time stamp indicating when such a switch will occur. Since the client can decide to switch, the client can know this information. However, the rendering server may need to transmit a PTS or other time stamp associated with a remotely rendered frame. This can be signaled via, for example, a WebSocket message, in the SEI message of a frame, or as an RTP sender report, and may include, for example, the following information:

[0219]

[0220] When a client switches to local rendering, it can discard the video stream or a portion of the video stream associated with the object identifier. When a client switches from local rendering to remote rendering, it can stop rendering the object locally when the received object identifier indicates that the video stream is for a remotely rendered version of the desired PTS containing the object.

[0221] Adaptive solution

[0222] When deciding between objects rendered locally on the client and objects rendered remotely by the rendering server, various standards can be used. In some cases, these standards can allow the client to adapt its hybrid rendering to certain events or situations. Such standards and corresponding adaptive schemes are described below.

[0223] Adaptive based on object complexity

[0224] Clients may have limited or even very limited processing power. The latter can be particularly relevant to certain types of mobile clients. Therefore, a distinction can be made between more complex objects and less complex or more regular objects, allowing the client to decide where to render which objects. Such complexity can be defined or indicated in various ways, including but not limited to the number of polygons and / or the number of nested object parts. The client's decision-making process regarding whether to render objects locally or remotely can, for example, involve:

[0225] 1. The client can determine the maximum number of polygons it can render.

[0226] 2. Scene descriptors can indicate the number of polygons for each object.

[0227] 3. The client can indicate to the remote rendering server the number of polygons it can render.

[0228] 4. The client can extract a remotely rendered video representation of a high-polygon object from the rendering server.

[0229] Examples of messages for remotely rendering complex objects may include the following information:

[0230]

[0231] Using the information provided in this message from the client, the rendering server can begin creating the requested video stream, which contains objects with a complexity exceeding the 'maximum complexity'. The rendering server can respond to the client's request using the following message:

[0232]

[0233] When a client wants to stop remote rendering of (multiple) objects, it can send the following message:

[0234]

[0235] Based on adaptive distance to the object

[0236] The decision of whether to render locally or remotely on the rendering server can additionally or alternatively depend on the client's location within the scene. Such location-dependent adaptation can be advantageous because rendering the entire scene on the client can be resource-intensive. By rendering only nearby objects, the client can conserve processing resources such as computing power or battery power. Continuing with the AR example, where a user starts at location A1 and moves to location A2, adaptation based on distance to objects could include the following:

[0237] 1. The client can extract the scene descriptor.

[0238] 2. The client can extract 3D assets.

[0239] 3. The client can determine its current location (A1).

[0240] a. Based on their location within the scene, the client can determine which objects are within its field of view and within a specific radius.

[0241] b. At location A1, this can result in the identification of zero objects, meaning that all other visible objects (e.g., outside a certain radius) can be rendered remotely by the rendering server.

[0242] c. Therefore, the client can extract a pre-rendered version of the video stream that includes these objects from the rendering server.

[0243] 4. The client can move to position A2.

[0244] a. The client can determine that object 1 is within its field of view and a specific radius, and therefore can render object 1 locally and combine the locally rendered object 1 with pre-rendered versions of the remaining visible objects from the rendering server.

[0245] Signaling which objects to render remotely can be done by following the same steps as switching between local and remote rendering, but this could require a long list of remotely rendered objects. Alternatively, such signaling could involve a message where the client signals which objects to render locally, and the rendering server can then determine which objects to render remotely based on the scene descriptor. The latter could involve determining which objects are visible to the client and omitting objects to be rendered locally.

[0246]

[0247] Using the information provided by this message, the rendering server can begin creating the requested video stream, which contains objects(s) not previously indicated for local rendering. The rendering server can respond to a client's request using the following message:

[0248]

[0249] After setting up remote rendering, the client can periodically, semi-periodically, or repeatedly send messages to the rendering server to update the client's position in the 3D environment, for example, using the following message, which may additionally include a field indicating an object identifier for objects to be rendered locally. The message may look like this:

[0250]

[0251] When a client wants to stop local rendering of (multiple) objects, it can send the following message to the rendering server. In response to this message, the rendering server can begin remotely rendering all relevant objects in the scene.

[0252]

[0253] Adaptive processing based on processing power

[0254] Rendering a scene can strain local processing resources (such as the client's CPU and GPU), especially for realistic scenes and so-called 6DOF (degrees of freedom) experiences. Using hybrid rendering, the client and the rendering server can share rendering tasks. The client can request the rendering server to assist with scene rendering in ways such as:

[0255] 1. Clients can monitor their CPU and / or GPU load.

[0256] 2. If the CPU and / or GPU reach 100% or any earlier threshold, the client may request (attached) remote rendering by the rendering server, for example, by remote rendering of attached (and / or high-poly) objects.

[0257] The rendering server can also initiate requests for assistance from the client:

[0258] 1. The server can monitor its CPU and / or GPU load and, for example, periodically transmit the CPU and / or GPU load to the client.

[0259] 2. The client can check whether the rendering server can handle the additional load associated with remotely rendered (attached) objects.

[0260] 3. If this additional load can be handled, the client can instruct the rendering server which(s) ...

[0261] Bandwidth-based adaptive

[0262] A client can estimate the bandwidth available for streaming, for example, by counting the number of bytes entering the network buffer and determining the average per second. Based on this estimated available bandwidth, the client can determine to switch to local rendering, remote rendering, or change the ratio in a hybrid rendering approach (e.g., more remote rendering or more local rendering). Specifically, if available bandwidth is limited, the client can switch or change the ratio to favor local rendering, while if sufficient available bandwidth exists, the client can switch or change the ratio to favor remote rendering. This could involve, for example, the following steps:

[0263] 1. If bandwidth is too low or reduced to zero (for example, when the client loses network connectivity), the client can switch to local rendering.

[0264] 2. If bandwidth is restored, the client can switch to remote rendering. In this case, a synchronization step can occur before the switch, for example, to verify that both the client and the rendering server have the same version of the scene description.

[0265] It will be understood that the client and / or rendering server may also monitor bandwidth in any other way and may switch between local and remote rendering, or change the ratio in hybrid rendering depending on such bandwidth measurements. Typically, if a client needs to render more objects locally, it can reduce the quality of its local rendering to be able to render those objects locally. Conversely, if the client is able to use remote rendering for at least some objects, it can again increase the quality of its local rendering.

[0266] Delay-based adaptive

[0267] In scenarios where the client can render at least some objects locally, latency can be considered to determine when or whether to switch to local rendering, remote rendering, or change the ratio in a hybrid rendering approach. For example, such latency can be defined as network latency between the client and the rendering server, or end-to-end latency, such as the time required for frame creation at the rendering server, encoding, encapsulation, transmission, and decoding at the client, or any other combination thereof. Typically, when utilizing remote rendering, frames may need to arrive 'on time,' meaning a time before the frame is to be displayed. Whether this is possible can depend on the latency of remote rendering. However, even if the latency is sufficient for timely frame reception and display, the client can still determine to switch fully or partially to local rendering to mitigate the chance of delayed or non-reception when network conditions deteriorate. For example, if network conditions worsen, the client might decide to remotely render more static objects while locally rendering more dynamic objects to reduce visual artifacts.

[0268] Latency can be calculated by the client. For example, when the client's clock is synchronized with the server's clock, the client can subtract the frame creation time from the time the frame was received by the client. The frame creation time can be indicated, for example, as a timestamp in the metadata accompanying the frame. The frame creation time can be signaled by the server via WebSocket, RTCP / RTSP commands, or it can be retrieved by the client via an HTTP REST API. The message can include, for example:

[0269]

[0270] The switching commands can be similar, such as switching between local and remote rendering, and descriptions of adaptive rendering based on distance to objects.

[0271] Adaptive based on battery level

[0272] Rendering a scene can require significant computing resources, which in turn can consume power. This can be detrimental to mobile clients. To conserve battery, clients can decide to partially or completely switch to remote rendering based on their battery level. This can lead to reduced power consumption, as rendering 3D graphics objects can be more power-intensive than receiving and decoding video streams. Such a switch can be implemented when the battery level drops below a threshold. In some embodiments, as the battery level decreases, the client can gradually request remote rendering of more objects. Additionally, or instead of battery level, the client can consider power consumption rate, as a higher power consumption rate can prompt the client to request remote rendering earlier than a lower power consumption rate. In addition to battery level, the client can also consider the complexity of the objects being rendered, as rendering highly complex objects can consume more power. For example, highly complex objects may be preferably rendered remotely, or the client may switch to remote rendering earlier if the scene contains more complex objects on average.

[0273] Combine remotely rendered objects with the local environment

[0274] Objects rendered remotely can be combined with objects rendered locally in various ways, examples of which are described below.

[0275] Any message can be transmitted in various ways (e.g., using WebSocket, in the SEI message of a frame, RTP sender report, etc.) or any of the previously mentioned message transmission methods.

[0276] Place video frames in a 3D environment

[0277] The client can indicate its position and orientation to the rendering server and receive a video-based representation of objects within its field of view. Conceptually, generating a video-based representation by the rendering server can be analogous to recording a scene by a virtual camera. The resulting video frames can be placed by the client into a 3D environment representing the scene at the indicated position. Such placement can represent a virtual screen in the 3D environment, onto which the output of the virtual camera can be projected. To achieve such placement, the rendering server can, for example, send the following information to the client for each frame or each keyframe of the video stream:

[0278]

[0279] Mix locally generated frames and remotely generated frames.

[0280] Another way to combine remotely rendered objects with locally rendered objects is to render each object as a corresponding frame and then overlay these frames before rendering them on the display. For example, two frames could simply be placed on top of each other, where at least one frame is partially transparent beyond its rendered content, or, if the video codec used does not support transparency, contains an opaque color that can be signaled to the client and made transparent by the client. To achieve this blending of frames, the rendering server could, for example, send the following information to the client for each frame or keyframe of the video stream:

[0281]

[0282] Receive video streams with 2D descriptions of 3D objects

[0283] Another example is when objects are rendered remotely, these objects can be transmitted as unfolded polygons in 2D frames. The client can reconstruct such objects by folding the polygons and placing the folded objects in a 3D environment with a specified texture. To achieve this client-side folding of unfolded polygons in 2D frames, the rendering server can send the following information to the client, for example, for each frame or each keyframe of the video stream:

[0284]

[0285] As mentioned earlier, the response can be a texture (or material, or other 2D content from a graphics engine). In such cases, anchor points (typically expressed as 2D positions) can be specified to indicate which point in the texture (e.g., top left or center) is the reference point, rather than position and rotation.

[0286] Reusable video containing objects

[0287] In some embodiments, each video stream received by the client may contain remotely rendered versions of one or more objects. The rendering server may generate a separate video stream for each of these objects or for each set of connected objects. Therefore, depending on the number of objects to be remotely rendered, the client may receive a large number of video streams, all of which may need to be decoded, for example, using a separate decoder or decoder instance. To limit the number of video streams transmitted to the client and therefore must be decoded by the client, the rendering server may multiplex two or more video streams into one video stream. In this case, the signaling to the client may include the following information:

[0288]

[0289] For this simple example, it is assumed that all packed objects in the frame are rectangular and have a known packing order. Other, more complex implementations may require additional information, such as anchor points, transparency maps, etc.

[0290] Scene graphics definition

[0291] A scene can be modeled at least in part by a scene graph. For example, a scene graph can be a directed acyclic graph, such as a naive tree structure representing an object-based hierarchy of the scene's geometry. Scene descriptor data can include such a scene graph or references to scene graphs stored elsewhere. Scene graphs can be defined in various ways, as will be illustrated below.

[0292] For example, depending on the distance from the viewpoint rendered by the client to one or more objects in the scene, it may be expected that objects are rendered locally (e.g., for nearby objects) or a video stream containing pre-rendered versions of objects may be received (e.g., for distant objects). Some objects in the scene may be completely or partially occluded (e.g., objects behind a window in an aquarium), or may be located in spaces inaccessible to the user (e.g., in a theater). These objects may form a group (e.g., a group of fish or a group of actors). In some cases, the group itself can be considered an object. There may also be objects that can include separate but connected object parts. In cases where objects include other objects or separate but connected object parts, it may not always be necessary to render such other objects or object parts separately at the client. Instead, in some cases, it may be advantageous to render these remotely rendered objects as a group of objects or object parts and deliver them to the client as a video stream.

[0293] In two scenarios, a group of objects can be considered as objects themselves, and objects comprising individual but connected object parts can be referred to 'nested objects' below, because a hierarchy can exist between objects or object parts. This hierarchy can be reflected in scene descriptor data, as scene descriptor data can include or reference a scene graph that can indicate the relationships between objects in the scene. Based on this scene graph, the client can decide whether to render objects locally and individually or retrieve a pre-rendered version of the video stream that includes the objects. The following can be an example of such a graph, where objects are defined as belonging to branches. For example, a branch can represent a group of objects, or a larger object that includes individual objects. Such graphs can be created in different formats, such as XML, JSON, plain text with indentation, etc. An example in XML format is shown below.

[0294]

[0295] In more complex cases, branches can also contain sub-branches. An example of this is shown below:

[0296]

[0297]

[0298] Message sending and receiving

[0299] When requesting the rendering server to render an object or part of an object that belongs to one or more branches of the scene graph, the following messages can be sent and received, followed by a message representing the server's response:

[0300]

[0301]

[0302] Typically, the embodiments described in this specification can have various variations. For example, in addition to defining the dynamic aspects of a scene, scene descriptor data may also include or reference content used in the dynamics. For example, scene descriptor data may include or reference animation data that defines the animation of an object, or interactive data that defines the changes of an object under certain conditions, or semantic properties of an object (e.g., “activity” or “artifact”), or post-processing metadata that defines rendering cues (e.g., for merging light or audio sources), content properties (e.g., “generated” or “captured”), and other information that may be useful to the client (e.g., object lifetime).

[0303] Typically, the content and syntax of the messages described in this specification can represent metadata defined elsewhere in this specification.

[0304] Typically, a scene can combine 3D graphical objects with video. For example, a 3D graphical object can represent a virtual display on which video can be displayed. A specific, but not limiting, example is a film with video avatars, where camera recordings of actors' faces are displayed as textures on the faces of 3D models of the actors. Additionally, when using hybrid rendering, some 3D graphical objects can be pre-rendered by a rendering server and delivered to the client as a video stream. From the client's perspective, a scene thus appears to include, for example, one or more video streams of 3D assets of remotely pre-rendered 3D graphical objects and 3D image objects to be rendered locally. In video streaming, such as when using MPEG DASH, the client can request a Media Rendering Descriptor File (MPD) and select one or more media segments to download. In a server-hosted 3D environment, the client can download a description of the 3D environment and all assets, and use that description and the downloaded assets to render the 3D environment. These two techniques can be combined to enable the rendering of scenes that combine 3D graphical objects with video. Therefore, a scene can be defined by two different types of descriptors: one or more MPDs and one or more 3D scene descriptors. By combining these two types of descriptors, the client can know where to retrieve the data it needs, such as one or more video streams and 3D assets of a scene.

[0305] In some embodiments, instead of the client determining where (e.g., locally or remotely) which objects are rendered, or otherwise, the rendering server may make this decision, partially or entirely, between local and remote rendering. In such embodiments, the client may send periodic updates to the server characterizing its local rendering conditions; these periodic updates may include information about battery level, connection strength, GPU usage, etc. The rendering server can then use this information to determine where and which objects are rendered. In such an example, the client may also be configured to switch to local rendering if the connection to the rendering server is poor or lost.

[0306] In some embodiments, at any given moment during a rendering session, the server may request the client to adjust the division between remote and local rendering, for example, to have fewer objects rendered remotely by the server if resources on the server are insufficient. Such a request may be ignored by the client.

[0307] In some embodiments, such as after a connection between the client and the rendering server is lost, scene verification can be triggered by the client or by the rendering server. Such scene verification may include, for example, comparing scene descriptor data between the client and the rendering server by exchanging hashes and / or version numbers of the scene descriptor data to check for consistency. If such verification fails, an update may be sent or the scene descriptor data may be retrieved again, for example, from a content server or from a corresponding entity believed to have valid or the latest version of the scene descriptor data.

[0308] Continue to refer to Figure 1 Regarding the rendering device 100, it should be noted that the network interface 120 of the rendering device 100 can be, for example, a wireless communication interface, also referred to as a radio interface, and can be configured to connect to mobile network infrastructure. In some examples, the network interface 120 may include a radio and an antenna, or include a radio and antenna connection. In specific examples, the network interface 120 may be a 4G or 5G radio interface for connecting to a 4G or 5G mobile network conforming to one or more 3GPP standards, or it may be a Wi-Fi communication interface for connecting to Wi-Fi network infrastructure, etc. In other examples, the network interface 120 may be a wired communication interface, such as an Ethernet-based or fiber-optic-based interface.

[0309] It should be noted that data communication between rendering device 100 and remote rendering system 200 may involve multiple networks. For example, rendering device 100 may be connected to the infrastructure of a mobile network via a radio access network and connected to the Internet via the mobile infrastructure, wherein remote rendering system 200 is a server that is also connected to the Internet.

[0310] The rendering device 100 may further include a processor subsystem 140, which may be configured, for example by hardware design or software, to perform operations described herein related to the rendering device or the rendering of a scene as an enhancement of a video object or external environment. Typically, the processor subsystem 140 may be embodied by a single central processing unit (CPU), such as an x86 or ARM-based CPU, but may also be embodied by a combination or system of such a CPU and / or other types of processing units, such as a graphics processing unit (GPU). The rendering device 100 may further include a display interface 180 for outputting display data 182 to a display 190. The display 190 may be an external or internal display of the rendering device 100, and is typically head-mounted or non-head-mounted. Using the display interface 180, the rendering device 100 can display the rendered scene as an enhancement of a video object or external environment. In some embodiments, the display 190 may include one or more sensors, such as accelerometers and / or gyroscopes, for detecting the user's posture. In such an embodiment, the display 190 may provide the sensor data 184 to the rendering device 100, for example, via the aforementioned display interface 180 or via a separate interface. In other embodiments, such sensor data 184 may be received separately from the display.

[0311] Also Figure 1 As shown, the rendering device 100 may include a data storage device 160 for storing data (including but not limited to scene descriptor data 20). The data storage device 160 may take various forms, such as a hard disk drive or hard disk drive array, a solid-state drive or solid-state drive array, a memory, etc.

[0312] Typically, rendering device 100 can be embodied by a (single) device or apparatus, such as a smartphone, personal computer, laptop computer, tablet device, game console, set-top box, television, monitor, projector, smartwatch, smart glasses, media player, media recorder, etc. In some examples, rendering device 100 can be a so-called user equipment (UE) of a mobile telecommunications network, such as 5G or next-generation mobile networks. In other examples, rendering device can be an edge node of the network, such as the aforementioned mobile telecommunications edge node. In such examples, the rendering client may lack display output, or may at least not use display output to display the rendered scene. Instead, the rendering device can render the scene, which can then be made available for streaming to another downstream rendering device, such as an end-user device.

[0313] Continue to refer to Figure 1Regarding the remote rendering system 200, it should be noted that the network interface 220 of the remote rendering system 200 can be, for example, a wired communication interface, such as an Ethernet-based or fiber-optic interface. The network 10 can be, for example, the Internet or a mobile network, wherein the remote rendering system 200 is connected to a fixed portion of the mobile network. Alternatively, the network interface 220 can be a wireless communication interface, for example, the type described above for rendering device 100.

[0314] The remote rendering system 200 may further include a processor subsystem 240, which may be configured, for example by hardware design or software, to perform the operations described herein related to the remote rendering system or generally to rendering a scene for a rendering device. Typically, the processor subsystem 240 may be embodied by a single CPU, such as an x86 or ARM-based CPU, but may also be embodied by a combination or system of such a CPU and / or other types of processing units, such as a GPU. In embodiments where the remote rendering system 200 is distributed across different entities (e.g., across different servers), the processor subsystem 240 may also be distributed across the CPUs and / or GPUs of such different servers. Also as Figure 1 As shown, the remote rendering system 200 may include a data storage device 260 such as a hard disk drive or hard disk drive array, a solid-state drive or solid-state drive array, or a memory, which may be used to store data including but not limited to scene description data 20.

[0315] The remote rendering system 200 can be distributed across various entities, such as local servers or remote servers. In some embodiments, the remote rendering system 200 can be implemented by a type of server or a system of such servers. For example, the remote rendering system 200 can be implemented by one or more cloud servers or by one or more edge nodes of a mobile network. In some embodiments, the remote rendering system 200 and the rendering device 100 can cooperate with each other according to a client-server model, wherein the rendering device 100 acts as a client.

[0316] Typically, each entity described in this specification may be embodied as a device or apparatus, or embodied in a device or apparatus. The device or apparatus may include one or more (micro)processors executing appropriate software. The processor(s) of the corresponding entity may be embodied by one or more of these (micro)processors. The software implementing the functionality of the corresponding entity may have been downloaded and / or stored in the corresponding one or more memories, for example, in volatile memory such as RAM or non-volatile memory such as flash memory. Alternatively, the processor(s) of the corresponding entity may be implemented in the device or apparatus in the form of programmable logic, for example, as a field-programmable gate array (FPGA). Any input and / or output interfaces may be implemented by the corresponding interfaces of the device or apparatus. Typically, each functional unit of the corresponding entity may be implemented in the form of a circuit or circuit system. The corresponding entities may also be implemented in a distributed manner, for example, involving different devices or apparatuses.

[0317] Figure 4 The steps of a computer-implemented method for blended rendering of a scene are shown, including steps performed at or by a local rendering device. Figure 4 The method may include: obtaining 510 scene descriptor data as described elsewhere in this specification; determining 520 a first portion of the scene to be rendered locally by a rendering device and a second portion of the scene to be rendered remotely, based at least on the scene descriptor data; and requesting 530 a remote rendering system authorized to access the scene descriptor data to render the second portion of the scene to obtain pre-rendered scene data, the pre-rendered scene data including a pre-rendered version of the second portion of the scene, wherein the request includes providing a timestamp to the remote rendering system to request the remote rendering system to render the second portion of the scene in a state corresponding to the timestamp. The method may further include: obtaining 540 the pre-rendered scene data from the remote rendering system; and rendering 550 the scene as an enhancement of a video object or external environment, rendering the first portion of the locally rendered scene and the pre-rendered version including the second portion of the scene.

[0318] Figure 5 The steps of a computer-implemented method for blended rendering of a scene are shown, as well as steps performed at or by a remote rendering system. Figure 5 The method may include: obtaining 610 scene descriptor data as described elsewhere in this specification; receiving 620 a request to render a portion of the scene, wherein the request includes a timestamp; rendering 630 a portion of the scene in a state corresponding to the timestamp based on the scene descriptor data and the request to obtain pre-rendered scene data, the pre-rendered scene data including a pre-rendered version of a portion of the scene; and providing 640 the pre-rendered scene data to a rendering device.

[0319] Note that any method described in this specification, such as any method described in any claim, can be implemented on a computer as a computer-implemented method, dedicated hardware, or a combination of both. Instructions for a computer (e.g., executable code) can be stored, for example... Figure 6 The computer-readable medium 700 shown may contain, for example, a series of machine-readable physical marks 710 and / or a series of elements having different electrical (e.g., magnetic) or optical properties or values. Executable code may be stored in a transient or non-transient manner. Examples of computer-readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Figure 6 An optical storage device 700 is illustrated by way of example.

[0320] In alternative embodiments, the computer-readable medium 700 may be a transient or non-transitory computer-readable medium and may include data 710 in the form of a data structure representing any scene descriptor data described in this specification.

[0321] Figure 7 This is a block diagram illustrating an exemplary data processing system 1000 that can be used in the embodiments described in this specification. Such a data processing system includes the data processing entities described in this specification, including but not limited to rendering devices and remote rendering systems.

[0322] The data processing system 1000 may include at least one processor 1002 coupled to a memory element 1004 via a system bus 1006. In this way, the data processing system can store program code within the memory element 1004. Furthermore, the processor 1002 can execute program code accessed from the memory element 1004 via the system bus 1006. In one aspect, the data processing system may be implemented as a computer suitable for storing and / or executing program code. However, it is understood that the data processing system 1000 may be implemented in the form of any system including a processor and memory capable of performing the functions described herein.

[0323] Memory element 1004 may include one or more physical memory devices, such as local memory 1008 and one or more mass storage devices 1010. Local memory may refer to random access memory or other non-persistent memory(s) typically used during the actual execution of the program code. Mass storage devices may be implemented as hard disk drives, solid-state drives, or other persistent data storage devices. Data processing system 1000 may also include one or more cache memories (not shown) that provide temporary storage for at least some program code to reduce the number of times program code is otherwise retrieved from mass storage device 1010 during execution.

[0324] Input / output (I / O) devices, depicted as input device 1012 and output device 1014, may optionally be coupled to the data processing system. Examples of input devices include, but are not limited to, microphones, keyboards, pointing devices such as mice, game controllers, Bluetooth controllers, VR controllers, and gesture-based input devices. Examples of output devices include, but are not limited to, monitors or displays, speakers, etc. Input and / or output devices may be coupled to the data processing system directly or via an intermediate I / O controller. Network adapter 1016 may also be coupled to the data processing system to enable it to be coupled to other systems, computer systems, remote network devices, and / or remote storage devices via an intermediate private or public network. The network adapter may include a data receiver for receiving data transmitted to and from the systems, devices, and / or networks, and a data transmitter for transmitting data to and from the systems, devices, and / or networks. Modems, cable modems, and Ethernet cards are examples of different types of network adapters that may be used with the data processing system 1000.

[0325] like Figure 7 As shown, memory element 1004 can store application program 1018. It is understood that data processing system 1000 can further execute an operating system (not shown) capable of facilitating the execution of the application program. The application program, implemented in the form of executable program code, can be executed by data processing system 1000 (e.g., by processor 1002). In response to executing the application program, the data processing system can be configured to perform one or more operations, which will be described in further detail herein.

[0326] For example, data processing system 1000 can be represented as shown in the reference. Figure 1 The rendering device described elsewhere in this specification. In this case, application 1018 may represent an application that, when executed, configures data processing system 1000 to perform the functions described in the reference rendering device. In another example, data processing system 1000 may represent the rendering device described in the reference... Figure 1 The remote rendering system described elsewhere in this specification. In this case, application 1018 may represent an application that, when executed, configures data processing system 1000 to perform the functions described with reference to the remote rendering system.

[0327] It should be noted that the embodiments mentioned above are illustrative and not limiting of the invention, and those skilled in the art will be able to devise many alternative embodiments without departing from the scope of the appended claims.

[0328] In the claims, any reference numerals placed between parentheses should not be construed as limiting the claims. The use of the verb "comprising" and its variations does not exclude the presence of elements or levels other than those stated in the claims. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. Expressions such as "at least one" preceding a list or group of elements indicate the selection of all elements or any subset of elements from the list or group. For example, the expression "at least one of A, B, and C" should be understood to include only A, only B, only C, both A and B, both A and C, both B and C, or all A, B, and C. The invention can be embodied by hardware comprising several different elements and by a suitably programmed computer. In an apparatus claim enumerating several means, several of these means can be embodied by the same hardware. The fact that certain measures are stated in mutually different dependent claims does not imply that combinations of these measures cannot be advantageously used.

Claims

1. A computer-implemented method for rendering a scene using a rendering device, wherein, The rendering device is configured to display video objects or allow the external environment to be visually conveyed to the viewer and render the scene as the video objects or an enhancement of the external environment. The method is executed through the rendering device. The method includes: - Obtain scene descriptor data, wherein the scene descriptor data identifies a set of 3D graphical objects representing at least a portion of the scene, wherein the scene descriptor data indicates the change of the state of the scene over time; - Based at least on the scene descriptor data, determine a first part of the scene to be rendered locally by the rendering device and a second part of the scene to be rendered remotely; - Determine a timestamp to indicate a second time instance, which precedes the first time instance in time in which the rendering device is currently rendering the scene; - A request is made to a remote rendering system authorized to access the scene descriptor data to render a second portion of the scene to obtain pre-rendered scene data, the pre-rendered scene data including a pre-rendered version of the second portion of the scene. The request includes providing the time stamp to the remote rendering system to request the remote rendering system to render a second part of the scene in a state corresponding to the time stamp; - Obtain the pre-rendered scene data from the remote rendering system; - Render the scene as an enhancement of the video object or the external environment, the rendering including a first portion of the scene rendered locally and a pre-rendered version of a second portion of the scene.

2. The computer-implemented method according to claim 1, wherein, The scene descriptor data describes the state of the scene at multiple time instances on the scene timeline to indicate how the state of the scene changes over time, and wherein the time marker indicates a time instance on the scene timeline.

3. The computer-implemented method according to claim 1 or 2, wherein, The scene descriptor data indicates at least one of the following: the presence, location, orientation, and appearance of a corresponding object in the scene as it changes over time.

4. The computer-implemented method of claim 1, further comprising determining the timestamp based on at least one of the following: - Network latency between the rendering device and the remote rendering system; - The latency or associated scheduling constraints of rendering the second part of the scene by the remote rendering system.

5. The computer-implemented method according to claim 1 or 2, wherein, The scene descriptor data includes the object identifier of the corresponding graphics object in the group of 3D graphics objects, wherein: - The request to the remote rendering system includes providing the remote rendering system with corresponding object identifiers for one or more 3D graphics objects representing a second part of the scene; and / or - Obtaining the pre-rendered scene data includes receiving metadata of corresponding object identifiers of one or more 3D graphics objects pre-rendered in the pre-rendered scene data.

6. The computer-implemented method according to claim 5, wherein, The method repeatedly receives the metadata, and further includes stopping or starting local rendering of the 3D graphics object based on the object identifier of the 3D graphics object that appears in or disappears from the metadata.

7. The computer-implemented method according to claim 1 or 2, wherein, The rendering of the scene can be adapted to the content of the video object or the external environment, and the method further includes determining content data that at least partially represents the content and providing the content data to the remote rendering system.

8. The computer-implemented method according to claim 7, wherein, The content data includes at least one of the following: - Illumination data characterizing the lighting in the content; - Depth data characterizing the depth of the content; and - Image or video data that provides a visual representation of the content.

9. The computer-implemented method according to claim 1 or 2, wherein, Determining the first portion of the scene to be rendered locally and the second portion of the scene to be rendered remotely includes determining whether to render the 3D graphics object locally or remotely based on at least one of the following: - The latency, bandwidth, or reliability of the network between the rendering device and the remote rendering system; - The latency, bandwidth, or reliability of the network between the rendering device and the content server, the content server being configured to host 3D graphics data of the group of 3D graphics objects that define the scene; - The computational load of the rendering device; - The computational load of the remote rendering system; - The battery level of the rendering device; - The complexity of the 3D graphical objects; as well as - The scene distance between the viewpoint of the rendering device when rendering the scene and the 3D graphics object in the scene.

10. The computer-implemented method according to claim 1 or 2, wherein, The scene descriptor data includes a graph containing branches indicating hierarchical relationships between 3D graphics objects, wherein determining a first portion of the scene to be rendered locally and a second portion of the scene to be rendered remotely includes determining whether each branch is to be rendered locally or remotely for the corresponding branch's 3D graphics object.

11. A computer-implemented method for rendering a portion of a scene using a rendering device, wherein, The rendering device is configured to display video objects or allow the external environment to be visually conveyed to the viewer and render the scene as the video objects or an enhancement of the external environment. The method is executed via a remote rendering system. The method includes: - Obtain scene descriptor data, wherein the scene descriptor data identifies a set of 3D graphical objects representing at least a portion of the scene, wherein the scene descriptor data indicates the change of the state of the scene over time; - Receive a request to render a portion of the scene, wherein the request includes a timestamp indicating a second time instance that precedes the rendering device in rendering a first time instance of the scene when the rendering device determines the timestamp; - Based on the scene descriptor data and the request, render a portion of the scene in a state corresponding to the time stamp to obtain pre-rendered scene data, the pre-rendered scene data including a pre-rendered version of a portion of the scene; - Provide the pre-rendered scene data to the rendering device.

12. The computer-implemented method of claim 11, further comprising generating metadata including corresponding object identifiers of one or more 3D graphics objects pre-rendered in the pre-rendered scene data, and providing the metadata together with the pre-rendered scene data to the rendering device.

13. The computer-implemented method of claim 11 or 12, further comprising adjusting a portion of the scene to be rendered by omitting 3D graphics objects from the rendering based on one of the following: - The computational load of the remote rendering system; - The latency, bandwidth, or reliability of the network between the rendering device and the remote rendering system.

14. A transient or non-transitory computer-readable medium comprising data representing a computer program, said computer program including instructions for causing a processor system to perform the method according to any one of claims 1 to 13.

15. A rendering device configured to display a video object or allow an external environment to be visually conveyed to a viewer and to render a scene including 3D graphics objects as an enhancement of the video object or the external environment. The rendering device includes: - Network interface with the network; - A data storage device including scene descriptor data, wherein the scene descriptor data identifies a set of 3D graphical objects representing at least a portion of the scene, wherein the scene descriptor data indicates the change of the state of the scene over time; - Processor subsystem, the processor subsystem being configured to: - Based at least on the scene descriptor data, determine a first part of the scene to be rendered locally by the rendering device and a second part of the scene to be rendered remotely; - Determine a timestamp to indicate a second time instance, which precedes the first time instance in time in which the rendering device is currently rendering the scene; - Through the network interface, a request is made to a remote rendering system authorized to access the scene descriptor data to render a second portion of the scene to obtain pre-rendered scene data, the pre-rendered scene data including a pre-rendered version of the second portion of the scene. The request includes providing the time stamp to the remote rendering system to request the remote rendering system to render a second part of the scene in a state corresponding to the time stamp; - Receive the pre-rendered scene data from the remote rendering system via the network interface; - Render the scene as an enhancement of the video object or the external environment, the rendering including a first portion of the scene rendered locally and a pre-rendered version of a second portion of the scene.

16. A rendering system, comprising: - Network interface with the network; - A data storage device including scene descriptor data, wherein the scene descriptor data identifies a set of 3D graphical objects representing at least a portion of the scene, wherein the scene descriptor data indicates the change of the state of the scene over time; - Processor subsystem, the processor subsystem being configured to: - Receive a request to render a portion of the scene via the network interface, wherein the request includes a timestamp indicating a second time instance that precedes the rendering device in rendering a first time instance of the scene when the rendering device determines the timestamp; - Based on the scene descriptor data and the request, render a portion of the scene in a state corresponding to the time stamp to obtain pre-rendered scene data, the pre-rendered scene data including a pre-rendered version of a portion of the scene; - The pre-rendered scene data is provided to the rendering device via the network interface.

Images