Method and apparatus for split rendering of light field / immersive media using a proxy edge cloud architecture

Abstract

Systems and methods for split rendering of immersive media using a proxy edge cloud computing architecture. The systems and methods may include dynamically determining a task split or splitting a task for rendering a scene in an immersive media stream into two or more computational tasks based on one or more processing delays, where the task split divides the scene into a first portion including one or more first tasks performed by an edge computing system and a second portion including one or more second tasks performed by a cloud computing system. The systems and methods may include transmitting the first portion of the scene and transmitting the second portion of the scene.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 284,956, filed December 1, 2021, and U.S. Patent Application No. 18 / 072,324, filed November 30, 2022, in the United States Patent and Trademark Office, the disclosures of which are incorporated herein by reference.

[0002] The disclosed subject matter relates to methods and systems for split rendering for immersive media. More particularly, the disclosed subject matter relates to methods and systems for split rendering for immersive media using a proxy edge cloud computing architecture. [Background technology]

[0003] Immersive media is defined by immersive technologies that attempt to create or mimic a physical world through digital simulation, thereby stimulating any or all human sensory systems to create the user's perception of being physically present inside the scene.

[0004] Various types of immersive media technologies are currently in use, including Virtual Reality, Augmented Reality (Artificial Intelligence), Mixed Reality (Magnetic Resonance), and Light Field / Holographic. VR refers to a digital environment that replaces a user's physical environment by placing them in a computer-generated world using a headset. VR, on the other hand, takes digital media and overlays them on the surrounding real world by using either clear vision glasses or a smartphone. MR refers to blending the real world with the digital world, thereby creating an environment where technology and the physical world can coexist.

[0005] Light field / holographic technology is composed of light rays in 3D space, with rays coming from each point and direction. The technology is based on the concept that everything observed is illuminated by light coming from an arbitrary light source, traveling through space, and hitting the surface of an object, where it is partially absorbed and partially reflected by another surface, before reaching our eyes. Which exact rays reach the eye depends on the user's exact position within the light field; as the user moves around, they perceive parts of the light field and use those perceived parts to get an idea of ​​the location of objects.

[0006] Rays can be defined by 5D plenoptic operations, where each ray can be defined by three coordinates in 3D space (three dimensions) and two angles to specify its direction in 3D space.

[0007] Conventional cameras can only capture a 2D representation of the light rays reaching the camera lens at a given position. The image sensor records the sum of the brightness and color of all the light rays reaching each pixel.

[0008] When capturing content for light field or holographic-based displays, a light field camera is required that is capable of capturing not only the brightness and color, but also the direction of all light rays reaching the camera sensor. Using this information, a digital scene can be reconstructed with an accurate representation of the origin of each ray, allowing the exact captured scene to be digitally reconstructed in 3D.

[0009] Currently, two main techniques are used to capture such volumetric scenes. The first technique involves using a camera or an array of camera modules to capture different light rays / views from each direction. The second technique consists of using a depth camera, which can capture 3D information in a single exposure without the need for structured lighting by measuring the depth of multiple objects under controlled lighting conditions. Summary of the Invention [Means for solving the problem]

[0010] The following presents a simplified summary of one or more embodiments of the present disclosure in order to provide a basic understanding of such embodiments. This summary is not an extensive overview of all contemplated embodiments, and is not intended to identify key or critical elements of all embodiments or to delineate the scope of any or all embodiments. Its sole purpose is to present some concepts of one or more embodiments of the present disclosure in a simplified form as a prelude to the more detailed description that is presented later.

[0011] According to an example embodiment, a method for split-rendering immersive media using a proxy edge-cloud computing architecture is provided. The method may be executed by one or more processors and may include: determining, by a rendering proxy server, that an end device consuming the immersive media has limited resource capacity; dividing, by the rendering proxy server, a task of rendering a scene in the immersive media stream into multiple tasks; dynamically grouping, by the rendering proxy server, the multiple tasks into two or more computational tasks based on one or more processing delays, where a first group of the two or more computational tasks is performed by an edge computing system and a second group of the two or more computational tasks is performed by a cloud computing system; transmitting, by the rendering proxy server, a first portion of the scene corresponding to the first group of the two or more computational tasks to the edge computing system for rendering the first portion of the scene; and transmitting, by the rendering proxy server, a second portion of the scene corresponding to the second group of the two or more computational tasks to the cloud computing system for rendering the second portion of the scene.

[0012] According to an example embodiment, an apparatus for split rendering of immersive media using a proxy edge cloud computing architecture is provided. The apparatus may include at least one memory configured to store program code and at least one processor configured to read the program code and operate as instructed by the program code. The program code may include: a first determination code configured to cause the at least one processor to determine, by a rendering proxy server, that an end device consuming the immersive media has limited resource capacity; a division code configured to cause the at least one processor to divide, by the rendering proxy server, a task of rendering a scene in the immersive media stream into multiple tasks; and a first grouping code configured to cause the at least one processor to dynamically group, by the rendering proxy server, the multiple tasks into two or more computational tasks based on one or more processing delays, the first group of the two or more computational tasks being computed by the edge computing system. the second group of two or more computational tasks is executed by the cloud computing system; a first sending code configured to cause the at least one processor to send, by the rendering proxy server, a first portion of the scene corresponding to the first group of two or more computational tasks to the edge computing system for rendering the first portion of the scene; and a second sending code configured to cause the at least one processor to send, by the rendering proxy server, a second portion of the scene corresponding to the second group of two or more computational tasks to the cloud computing system for rendering the second portion of the scene.

[0013] According to an example embodiment, a non-transitory computer-readable medium may be provided that stores instructions. The instructions may include one or more instructions that, when executed by one or more processors of a device for split-rendering immersive media using a proxy edge-cloud computing architecture, cause the one or more processors to: determine that an end device consuming the immersive media has limited resource capacity; split a task of rendering a scene in the immersive media stream into multiple tasks; dynamically group the multiple tasks into two or more computational tasks based on one or more processing delays, where a first group of the two or more computational tasks is performed by an edge computing system and a second group of the two or more computational tasks is performed by a cloud computing system; send a first portion of the scene corresponding to the first group of the two or more computational tasks to the edge computing system for rendering the first portion of the scene; and send a second portion of the scene corresponding to the second group of the two or more computational tasks to the cloud computing system for rendering the second portion of the scene.

[0014] Additional embodiments will be set forth in the description that follows, and in part will be obvious from the description, and / or may be learned by practice of presented embodiments of the present disclosure.

[0015] The foregoing and other objects, features, and advantages of the present disclosure will become apparent from the following detailed description taken in conjunction with the accompanying drawings in which like parts are given like reference numerals and in which: [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 illustrates an exemplary block diagram of a proxy edge cloud computing architecture for dynamically splitting and rendering immersive media, according to one embodiment of the present disclosure. [Figure 2]1 illustrates an exemplary flowchart of a proxy edge cloud computing architecture for dynamically splitting and rendering immersive media, according to one embodiment of the present disclosure. [Figure 3] 1 illustrates an exemplary flowchart of a proxy edge cloud computing architecture for dynamically splitting and rendering immersive media, according to one embodiment of the present disclosure. [Figure 4] FIG. 1 is a simplified block diagram of a communication system according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0017] The images in the drawings are simplified for illustrative purposes and are not drawn to scale. Within the description of the figures, similar elements are given similar names and reference numbers as those in the previous figures. The specific numbers assigned to elements are provided merely to aid in explanation and are not intended to suggest any limitations (structural or functional) on the invention.

[0018] The following detailed description of the exemplary embodiments refers to the accompanying drawings, in which the same reference numbers in different drawings may identify the same or similar elements.

[0019] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit implementations to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations. Furthermore, one or more features or components of one embodiment may be incorporated into or combined with another embodiment (or one or more features of another embodiment). Additionally, in the flowcharts and descriptions of operations provided below, it should be understood that one or more operations may be omitted, one or more operations may be added, one or more operations may occur (at least partially) concurrently, or the order of one or more operations may be rearranged.

[0020] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limiting of the implementation. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0021] Although particular combinations of features are recited in the claims and / or disclosed herein, these combinations do not limit the disclosure of possible implementations. Indeed, many of these features can be combined in ways not specifically recited in the claims and / or disclosed herein. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible implementations includes each dependent claim in combination with all other claims in the claim set.

[0022] No element, act, or instruction used herein should be construed as critical or required unless explicitly described as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used synonymously with "one or more." Where only one item is intended, the term "one" or similar language is used. Also, as used herein, the terms "has," "have," "having," "include," "including," etc. are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless otherwise specified. Furthermore, phrases such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood to include A only, B only, or both A and B.

[0023] Throughout this specification, references to "one embodiment," "an embodiment," or similar language mean that a particular feature, structure, or characteristic described in connection with the illustrated embodiment is included in at least one embodiment of the solution. Thus, throughout this specification, the phrases "in one embodiment," "in an embodiment," and similar language may, but do not necessarily, all refer to the same embodiment.

[0024] Furthermore, the described features, advantages, and characteristics of the present disclosure may be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize, in light of the description herein, that the present disclosure may be practiced without one or more of the specific features or advantages of a particular embodiment. In other cases, additional features and advantages may be recognized in particular embodiments that may not be present in all embodiments of the present disclosure.

[0025] Embodiments of the present disclosure are directed to systems and methods for split rendering for light field or immersive media through the use of a rendering proxy, an edge cloud, and a cloud-based architecture, which may be referred to herein as a proxy edge cloud architecture.

[0026] In the prior art, all computing devices and their performance depend on increasing computer power. Following Moore's Law, processing power continues to increase exponentially as a result of adding more processing cores and bandwidth, as well as increasing speed and decreasing chip size. However, high-performance applications place increasing demands on capacity and processing requirements. To bridge this gap, an edge cloud-based rendering architecture may be needed.

[0027] Cloud-based computing provides end users with more ways to access the massive amounts of computing power required for modern computer graphics. With the current debate over how much processing one can fit into a smartphone or any AR / VR device to render high-quality video frames, the shift to cloud rendering is improving the rendering of high-quality video frames. Thus, if a fast enough connection to significantly more powerful online computing resources could be opened, any small device could become a supercomputer capable of streaming real-time video and games.

[0028] Edge cloud-based split-rendering architectures can be used for light field / immersive media streaming, and although edge cloud-based split-rendering architectures reduce the amount of local processing power required, not all user devices have this reduced amount of local processing power. To bridge this gap, a proxy edge cloud split-rendering architecture is needed. This uses a rendering proxy on behalf of end devices in edge cloud and other architectures, and the rendering proxy transcodes or converts the rendered scene into a video format that is more easily processed on devices with limited local processing power.

[0029] Embodiments of the present disclosure are directed to split rendering for light field / immersive media using a proxy edge-cloud architecture that enables mobile devices with lower processing capabilities / power to render and stream light field / immersive media.

[0030] In one embodiment, a proxy edge-cloud-based split-rendering architecture can be used for light field / immersive media streaming. This reduces the requirements for local processing on the end device itself. For example, the end device may not require a GPU to provide an acceptable user experience. The task division between the edge and the cloud may be dynamic, i.e., the task division between the edge and the cloud may be based on factors such as sampling delay, computation delay including image processing and frame rendering delay, and networking delay including queuing and transmission delay.

[0031] FIG. 1 is an exemplary block diagram of a proxy edge cloud computing architecture 100 for dynamically splitting and rendering immersive media, according to one embodiment of the present disclosure.

[0032] Referring to Figure 1, a scene 101 can be divided into three computational tasks (also referred to as components or parts of the scene 102, 103, 104): Rendering Proxy 108 streams scene 1 107 and scene 3 106 from the cloud 105 (also referred to as a cloud computing system) and streams scene 2 109 from the edge 110 (also referred to as an edge computing system). For the avoidance of doubt, references herein to "scenes" are by way of example only and it should be understood that "scene" includes any media amenable to streaming.

[0033] According to one aspect of the present disclosure, when the rendering proxy 108 receives scenes from the cloud 105 and the edge 110, the rendering proxy 108 transcodes or converts these scenes into a less computationally intensive format and sends this converted video 112 to the final end device 111, which renders the converted video for the end user.

[0034] In some embodiments, the rendering proxy 108 can determine the capabilities of the end device and then apply the appropriate transcoding technique for the end device. In some embodiments, the rendering proxy 108 can dynamically change the transcoding technique or transcoding parameters used on behalf of the end device based on feedback from the end device. In some embodiments, the rendering proxy 108 can transcode some scenes as they are received from the cloud 105 and the edge 110 and forward other scenes to the end device without conversion. In some embodiments, the rendering proxy 108 can change its decision to transcode some scenes and forward other scenes to the end device based on feedback from the end device.

[0035] FIG. 2 illustrates an example flowchart of a process 200 for dynamically split-rendering immersive media using a proxy edge cloud computing architecture, according to one embodiment of the present disclosure.

[0036] 2, in operation 205, a processor of the rendering proxy server system may dynamically divide the task or determine a task division for rendering a scene in the immersive media stream into two or more computational tasks based on one or more processing delays. In some embodiments, the task division may divide the scene into one or more first tasks performed by the edge computing system and one or more second tasks performed by the cloud computing system. As an example, a processor of the rendering proxy 108 server system may dynamically determine a task division for rendering a scene 101 in the immersive media stream into two or more computational tasks (102, 103, 104) based on one or more processing delays.

[0037] In some embodiments, the processing delay includes one or more of a sampling delay, a computation delay, an image processing load, a frame rendering delay, and a networking delay, the networking delay further including queuing and transmission delay.

[0038] In operation 210, a processor of the rendering proxy server system can receive one or more first portions of a scene from a first processor of the edge computing system, where the one or more first portions of the scene can be based on one or more first tasks performed by the edge computing system. As an example, a processor of the rendering proxy server 105 system can receive one or more first portions of a scene 102 from a first processor of the edge computing system 110.

[0039] In operation 215, a processor of the rendering proxy server system can receive one or more second portions of the scene from a second processor of the cloud computing system, where the one or more second portions of the scene can be based on one or more second tasks performed by the cloud computing system. As an example, a processor of the rendering proxy server 105 system can receive one or more second portions of the scenes 103 and 104 by the second processor of the cloud computing system 105.

[0040] At operation 220, a processor of the rendering proxy server system may transmit one or more portions of the scene. In some embodiments, the transmitted scene may be converted or transcoded based on end device capabilities. In some embodiments, the transmitting may include the rendering proxy determining end user device capabilities associated with the end user receiving the immersive media stream and converting one or more portions of the scene to a computationally demanding format based on the end user device capabilities.

[0041] In some embodiments, the conversion is based on end-user device capabilities. In some embodiments, the end-user device capabilities are based on the type of end-user device or the processing capabilities of the end-user device. In some embodiments, the processor of the rendering proxy 108 server system can determine the capabilities of the end-user device (e.g., end device 111) and then apply the appropriate transcoding technique for the end-user device. In some embodiments, the processor of the rendering proxy 108 server system can dynamically change the transcoding technique or transcoding parameters used on behalf of the end-user device based on feedback from the end-user device. In some embodiments, the processor of the rendering proxy 108 server system can transcode some scenes and forward other scenes to the end-user device without conversion as received from the cloud 105 and edge 110. In some embodiments, the processor of the rendering proxy 108 server system can change its decision to transcode some scenes and forward other scenes to the end-user device based on feedback from the end-user device.

[0042] In operation 225, a processor of the rendering proxy server system can transmit the scene including the first and second portions of the scene to the end user device. As an example, a processor of the rendering proxy 108 server system can transmit the scene including one or more transformed portions of the scene or the second portion of the scene to the end device 111.

[0043] According to one aspect of the present disclosure, process 200 may include additional operations performed by a processor of the rendering proxy server system. The additional operations may include receiving one or more second portions of the second scene from a second processor of the cloud computing system, transcoding the one or more second portions of the second scene, and transmitting a second scene including the one or more transcoded second portions of the second scene and the one or more first portions of the second scene.

[0044] FIG. 3 illustrates an example flowchart of a process 300 for dynamically split-rendering immersive media using a proxy edge cloud computing architecture, according to one embodiment of the present disclosure.

[0045] 3, in operation 305, a processor of the rendering proxy server system can transmit a scene including one or more first and second portions of the scene to an end user device. As an example, a processor of the rendering proxy 108 server system can transmit a scene including one or more transformed portions of the scene to an end device 111.

[0046] At operation 310, a processor of the rendering proxy server system may receive feedback related to the transmitted scene. At operation 315, the processor of the rendering proxy server system may dynamically divide or determine a task division for rendering a second scene in the immersive media stream into two or more second computational tasks based on one or more processing delays and the feedback. Then, at operation 320, the processor of the rendering proxy server system may transform and / or transmit one or more portions of the second scene.

[0047] It can be understood that actions 205-225 of process 200 and actions 305-320 of process 300 can be performed in any order and / or combined.

[0048] The techniques for split rendering for light field or immersive media using a proxy edge cloud architecture disclosed herein may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 4 illustrates a computer system 400 suitable for implementing certain embodiments of the disclosed subject matter.

[0049] Computer software may be coded using any suitable machine code or computer language that may undergo assembly, compilation, linking, or similar mechanisms to create code comprising instructions that may be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc., directly, or through interpretation, microcode execution, etc.

[0050] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0051] 4 for computer system 400 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system 400.

[0052] Computer system 400 may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users, for example, through tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). Human interface devices may also be used to capture certain media that do not necessarily involve direct conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0053] The input human interface devices may include one or more of a keyboard 401, a mouse 402, a trackpad 403, a touchscreen 410, a data glove (not shown), a joystick 405, a microphone 406, a scanner 407, and a camera 408 (only one of each is shown).

[0054] The computer system 400 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses through, for example, tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen 410, data gloves (not shown), or joystick 405, although some haptic feedback devices may not function as input devices), audio output devices (such as speakers 409, headphones (not shown)), visual output devices (such as screens 410, including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input capability and each with or without haptic feedback capability, some of which may be capable of outputting output in more than three dimensions through means such as two-dimensional visual output or stereo output), virtual reality glasses (not shown), holographic displays and smoke tanks (not shown), and printers (not shown).

[0055] The computer system 400 may also include human-accessible storage devices and their associated media, such as optical media including a CD / DVD ROM / RW 420 having media 421 such as CDs / DVDs, thumb drives 422, removable hard drives or solid state drives 423, legacy magnetic media such as tape and floppy disks (not shown), and dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0056] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0057] The computer system 400 may also include interfaces to one or more communication networks. The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide-area, metropolitan, vehicular, and industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks such as Ethernet; cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, and the like; TV wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; vehicular and industrial networks including CANBus; and the like. Particular networks generally require external network interface adapters attached to particular general-purpose data ports or peripheral buses (449) (e.g., ports on the computer system 400); others are generally integrated into the core of the computer system 400 by attachment to a system bus as described below (e.g., Ethernet interface 435 to a personal computer system, or cellular network 433 interface to a smartphone computer system). Using any of these networks, the computer system 400 can communicate with other entities. Such communications may be one-way receive only (e.g., broadcast TV), one-way transmit only (e.g., CANbus to a particular CANbus device), or two-way to other computer systems, for example, using local or wide area digital networks. Specific protocols and protocol stacks may be used over each of these networks and network interfaces, as described above.

[0058] The aforementioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core 440 of the computer system 400 .

[0059] The core 440 may include one or more central processing units (CPUs) 441, graphics processing units (GPUs) 442, dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) 443, hardware accelerators 444 for specific tasks, etc. These devices may be connected via a system bus 448, along with read-only memory (ROM) 445, random access memory 446, and internal mass storage 447, such as a non-user-accessible internal hard drive or SSD. In some computer systems, the system bus 448 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected to the core's system bus 448 directly or via a peripheral bus 449. Peripheral bus architectures include PCI, USB, etc.

[0060] The CPU 441, GPU 442, FPGA 443, and accelerator 444 may combine to execute specific instructions that may constitute the aforementioned computer code, which may be stored in ROM 445 or RAM 446. Transient data may also be stored in RAM 446, while persistent data may be stored, for example, in internal mass storage 447. Fast storage and retrieval from any memory device may be enabled through the use of cache memory, which may be closely associated with one or more of the CPU 441, GPU 442, mass storage 447, ROM 445, RAM 446, etc.

[0061] The computer-readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0062] By way of example and not limitation, architecture 400, and specifically a computer system having core 440, can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage, as introduced above, as well as media associated with specific storage of core 440 that is non-transitory in nature, such as core internal mass storage 447 or ROM 445. Software implementing various embodiments of the present disclosure may be stored within such devices and executed by core 440. Computer-readable media may include one or more memory devices or chips, depending on particular needs. The software can cause core 440, and specifically the processors therein (including a CPU, GPU, FPGA, etc.), to perform particular processes or particular portions of particular processes described herein, including defining data structures stored in RAM 446 and modifying such data structures according to software-defined processes. Additionally, or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 444) that may operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. References to software may encompass logic, where appropriate, and vice versa. References to computer-readable media may encompass circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0063] While this disclosure describes several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope. [Explanation of symbols]

[0064] 100 Proxy Edge Cloud Computing Architecture 101 scenes 102 Scene 1 103 Scene 2 104 Scene 3 105 Cloud 106 Scene 3 107 Scene 1 108 Rendering Proxy 109 Scene 2 110 Edge 111 End Device 112 converted videos 400 Computer Systems 401 Keyboard 402 Mouse 403 Trackpad 405 Joystick 406 Microphone 407 Scanner 408 Camera 410 Touchscreen 420 CD / DVD ROM / RW 421 Medium 422 thumb drive 423 Removable Hard Drive or Solid State Drive 430 graphics adapter 433 Cellular Network 435 Ethernet interface 440 cores 441 Central Processing Unit (CPU) 442 Graphics Processing Unit (GPU) 443 Field Programmable Gate Area (FPGA) 444 Hardware Accelerator 445 Read-Only Memory (ROM) 446 Random Access Memory 447 Internal Mass Storage 448 System Bus 449 Peripheral Bus

Claims

1. 1. A method for split rendering of immersive media using a proxy edge cloud computing architecture including an edge computing system, a cloud computing system, an end device, and a rendering proxy server system interposed between the edge computing system and the end device, and between the cloud computing system and the end device, the method being executed by one or more processors of the rendering proxy server system, the method comprising: determining that an end device consuming the immersive media has limited rendering capabilities; If the end device consuming the immersive media is determined to have limited rendering capabilities, dividing a task of rendering a scene in an immersive media stream into multiple tasks; dynamically grouping the plurality of tasks into two or more computational tasks based on one or more processing delays, including processing delays associated with rendering operations of the end device, wherein a first group of the two or more computational tasks is performed by the edge computing system and a second group of the two or more computational tasks is performed by the cloud computing system; transmitting the first portion of the scene corresponding to the first group of the two or more computational tasks to the edge computing system for rendering the first portion of the scene; transmitting the second portion of the scene corresponding to the second group of the two or more computational tasks to the cloud computing system for rendering the second portion of the scene; A method comprising:

2. 2. The method of claim 1, wherein the one or more processing delays include one or more of a sampling delay, a computation delay, an image processing load, a frame rendering delay, and a networking delay, the networking delay further including a queuing and transmission delay.

3. The method comprises: receiving the first portion of the scene corresponding to the first group of the two or more computational tasks based on the first group of the two or more computational tasks being executed by the edge computing system; receiving the second portion of the scene corresponding to the second group of the two or more computational tasks based on the second group of the two or more computational tasks being executed by the cloud computing system; The method of claim 1 further comprising:

4. The step of transmitting the first portion of the one scene comprises: determining a processing capability of the end device that consumes the immersive media; converting the first portion of the one scene into a less computationally demanding format based on the processing capabilities; 2. The method of claim 1, comprising:

5. The method comprises: receiving, by the rendering proxy server system, feedback from the end device regarding processing delays of the end device associated with the first portion of the scene and the second portion of the scene; dividing a second task of rendering a second scene in the immersive media stream into a plurality of second tasks; dynamically grouping the plurality of second tasks into two or more second computational tasks based on feedback regarding the one or more processing delays and processing delays of the end device, wherein one computational task of the two or more second computational tasks corresponds to a first portion of the second scene that is not transcoded and another computational task of the two or more second computational tasks corresponds to a second portion of the second scene that is transcoded; The method of claim 1 further comprising:

6. The method of claim 1 , wherein transmitting the second portion of the one scene comprises converting and transmitting the second portion of the one scene.

7. The method comprises: receiving a first portion of the second scene from a first processor of the edge computing system; receiving a second portion of the second scene from a second processor of the cloud computing system; transforming the second portion of the second scene; transmitting the second scene including the second portion of the second scene and the first portion of the second scene; 6. The method of claim 5, further comprising:

8. Apparatus configured to carry out the method of any one of claims 1 to 7.

9. A computer program product for causing one or more processors to carry out the method of any one of claims 1 to 7.

Images