Asset Reusability in Light Field / Holographic Media

Abstract

Methods and apparatus for controlling light field and / or holographic media streamed to a client, determining bandwidth based on a state of a client buffer, splitting a light field and / or holographic media scene, determining whether assets are present in the client buffer, mapping the assets to respective ones of the split scenes and to the client buffer, controlling transmission of assets to the client buffer based on the bandwidth, the mapping, and whether one or more of the assets are determined to be present in the client buffer, and controlling the client to reuse at least one of the assets when rendering at least one of the split scenes.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 275,789, filed November 4, 2021, and U.S. Application No. 17 / 970,035, filed October 20, 2022, the entire contents of which are incorporated herein by reference.

[0002] The disclosed subject matter relates to methodologies for storing assets in a client buffer for reusability in light field and / or holographic displays. [Background technology]

[0003] Immersive media is defined by immersive technologies that attempt to create or mimic the physical world through digital simulation, thereby simulating any or all of the human sensory systems to create the perception of the user being physically present in the scene.

[0004] There are various types of current immersive media technologies, such as Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), and Light Field / Holographic. VR refers to a digital environment that replaces the user's physical environment by placing the user in a computer-generated world using a headset. AR, on the other hand, takes digital media and layers them onto the real world around the user, either through the use of clear vision, headsets, glasses, etc., or a smartphone. MR refers to the blending of the real and digital worlds, thereby creating an environment where technology and the physical world can coexist.

[0005] Light field and / or holographic technology is composed of light rays in 3D space with rays coming from each point and direction. It is based on the concept that everything we see around us is illuminated by light coming from any source, moving through space and hitting the surfaces of objects where the light is partially absorbed and partially reflected to another surface before reaching the eye. By properly recreating the light field, the user can obtain 3D effects such as binocularity and continuous motion parallax. The basic concept behind light field and / or holographic displays is a large array of projection modules that recreate an approximation of a light field and / or holographic by projecting light rays onto a holographic screen to show different but consistent information in slightly different directions.

[0006] The rays may be defined by a five-dimensional plenoptic function, and each ray may be defined by three coordinates in 3D space (the third dimension) and two angles to specify a direction in the 3D space.

[0007] To capture 360-degree video content, a 360-degree camera is required, but when it comes to capturing light field and / or holographic display content, expensive setups with multiple depth cameras or arrays of cameras are required, depending on the FoV of the scene to be rendered.

[0008] A conventional camera can only capture a 2D representation of the light rays reaching the camera lens at a given position: the image sensor records the sum of the brightness and color of all the light rays reaching each pixel.

[0009] When it comes to capturing the content of light field and / or holographic displays, what is needed is a light field camera that can capture not only the brightness and color, but also the direction of all the light rays that reach the camera sensor. Using this information, a digital scene can be reconstructed with an accurate representation of the origin of each light ray, making it possible to digitally reconstruct the scene accurately captured in 3D.

[0010] Currently, two main techniques are used to capture such volumetric scenes. The first of these techniques uses an array of cameras or camera modules to capture a different ray / view from each direction, respectively into one of the cameras of the array or one of the camera modules. The second of these techniques uses a depth camera that can capture 3D information in a single exposure, without the need for structured illumination, by measuring the depth of multiple objects under controlled lighting conditions.

[0011] Multimedia content for such light field and / or holographic displays is captured and stored on a server. This can be real-world or synthetic content. To transmit this data to the end client, a large amount of bandwidth is required, even after the data is compressed. Thus, in situations where bandwidth is limited, the client may experience buffering or interruptions, making the experience unpleasant. This is the technical problem addressed by the technical solutions described throughout this application.

[0012] That is, existing computer technologies are technically inadequate due to data storage and transmission constraints that may be experienced from large bandwidth requirements in the context of processing multimedia content for light field and / or holographic displays. Thus, technical solutions to such problems are desired. Thus, the present application discloses addressing that computer technology by improving asset storage and transmission capabilities faced by buffers, such as client buffers, via reusability in either light field and / or holographic displays. Summary of the Invention [Means for solving the problem]

[0013] The present invention relates to a method for streaming a light field and / or holographic media scene to a client, the method comprising: a memory configured to store computer program code; and one or more hardware processors configured to access the computer program code and operate as instructed by the computer program code, the computer program code comprising: a control code configured to cause the at least one hardware processor to control a light field and / or holographic media scene that is streamed to a client; a determination code configured to cause the at least one hardware processor to determine a bandwidth available to a client for streaming the light field and / or holographic media based at least on a state of a client buffer associated with the client; and a partitioning code configured to cause the at least one hardware processor to partition the light field and / or holographic media scene into a plurality of split scenes, each split scene including one or more assets of the scene. and a hardware processor including a processor for rendering at least one of the split scenes, the processor including a processor for reusing at least one of the assets, and further decision code configured to cause the at least one hardware processor to determine whether one or more assets are present in the client buffer; mapping code configured to cause the at least one hardware processor to map one or more assets of the scene to a respective split scene based on whether the one or more assets are determined to be present in the client buffer; transmission control code configured to cause the at least one hardware processor to control transmission of a subset of the one or more assets to the client buffer based on the bandwidth, the mapping, and whether the one or more assets are determined to be present in the client buffer; and reusability control code configured to cause the at least one hardware processor to control a client to reuse at least one of the assets when rendering at least one of the split scenes.

[0014] According to an exemplary embodiment, the at least one hardware processor is in network communication with a server with clients.

[0015] According to an exemplary embodiment, the computer program code further includes tracking code configured to cause the at least one hardware processor to track which assets are sent to a client device for each preceding scene and subsequent scene of the split scene.

[0016] According to an exemplary embodiment, the at least one hardware processor is in the client.

[0017] According to an example embodiment, controlling the transmission of one or more assets of the plurality of assets to the client buffer is further based on a relative depth of the asset within the light field and / or holographic media scene.

[0018] According to an exemplary embodiment, controlling the transmission of one or more assets of the plurality of assets to the client buffer is further based on a relative priority of the assets within the light field and / or holographic media scene.

[0019] According to an exemplary embodiment, determining whether one or more of the assets are present in a client buffer further includes determining whether one or more of the assets are present in any of a plurality of client buffers, each of which includes a client buffer that is network-connected over a local area network (LAN).

[0020] According to an exemplary embodiment, the computer program code further includes assignment code configured to cause the at least one hardware processor to assign a reusability priority to the at least one asset based on at least one of the number of times the at least one asset appears in the scene.

[0021] According to an exemplary embodiment, controlling the client to reuse at least one of the assets when rendering at least one of the split scenes is based on a reusability priority assigned to at least one of the assets.

[0022] According to an example embodiment, assigning a reusability priority to the at least one of the assets is further based on at least one of a size of the at least one of the assets and a complexity of the at least one of the assets. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is a diagram illustrating depth-based streaming according to example embodiments described herein. [Diagram 2] FIG. 2 is a diagram illustrating asset priority based streaming according to an example embodiment described herein. [Diagram 3] FIG. 2 illustrates a server controlled buffer according to an example embodiment described herein. [Figure 4] FIG. 2 illustrates a client controlled buffer according to an example embodiment described herein. [Diagram 5] FIG. 1 illustrates a computer system in accordance with an example embodiment described herein. [Figure 6] FIG. 1 is a flow diagram according to an example embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] The proposed features described below may be used individually or combined in any order. Furthermore, the embodiments may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium. In this disclosure, a most probable mode (MPM) can refer to a primary MPM, a secondary MPM, or both a primary MPM and a secondary MPM.

[0025] Light field and / or holographic techniques aim to create a virtual environment with accurate depth perception and three-dimensionality without the need to use any headset, thus avoiding side effects such as sickness. The server can use depth-based adaptive streaming for light field and / or holographic displays. Here, according to an exemplary embodiment, instead of rendering the entire scene at once, a bandwidth-based depth approach is considered. If the network capacity is relatively ideal, the end client can receive and render the entire scene at once. However, if there is bandwidth limitation, the end client renders the scene to a specific depth instead of rendering the entire scene. This depth is a function of the client's bandwidth. Thus, after obtaining information about the end client's bandwidth, the server adjusts the media being streamed between scenes with various depths. Referring to FIG. 1 showing an example 100 of depth-based streaming, object 101, object 102, and object 103 are different objects in one scene at varying distances 107, 106, and 105, respectively, from a camera 104 that can represent the user's viewpoint. After acquiring media, such as in S601 of the example 600 of FIG. 6, the server may assign a priority value to each asset in S602, which may also be used for adaptive streaming of light field and / or holographic displays. Here, according to an exemplary embodiment, instead of rendering the entire scene at once, a bandwidth-based asset priority approach is considered. If the network capacity is not limited, the end client may receive and render the entire scene assets at once. However, if there is a bandwidth limit as determined in S603, the end client may render assets with higher priority instead of rendering all assets in the scene by determining an adjustment in S604 and considering the priority in S606. Thus, the total assets rendered are a function of the client's bandwidth.Referring to FIG. 2 showing an example of asset priority based streaming 200, asset 201, asset 202, and asset 203 are different assets at various distances, and in S605, it can be determined whether to take into account the rendering distance and / or in S606, whether to take into account pre-set object priorities at priorities 1, 3, and 2, respectively, for transmission to and / or rendering at the client from camera 204, which may be the user's viewpoint.

[0026] In one embodiment, while a scene is being streamed, assets may be kept in a client's buffer for reuse in successive scenes. Assets may be added or removed from the client's buffer, such as at S610, based on different parameters as described below according to an exemplary embodiment.

[0027] In one embodiment, asset mapping is performed once the light field and / or holographic scene is split into multiple scenes. This is done by tracking the assets transmitted in the previous scene and transmitting only the assets of the successive scene that are not present in the previous scene. Asset control may be performed by the server or the client depending on the method used according to the exemplary embodiment.

[0028] In the same or another embodiment, when the asset buffer is controlled by the server, the server tracks the assets sent in the previous and subsequent scenes while splitting the scene into multiple scenes. Thus, when the first scene is sent by the server, the server tracks the assets that are in the client's buffer by checking in S607, and only sends in S608 the assets that are not already in the client's buffer, and does not send in S609 the other assets that are already in the buffer. The server also maintains the assets in the client's buffer so that the addition and removal of assets is managed by the server. For example, referring to FIG. 3 showing an example 300 of server-controlled buffering, the server 301 splits a scene 311 into two scenes 307 and 308 with respective assets "scene 1" 309 and "scene 2" 310. The client 302 streams "scene 1" 304 and "scene 2" 303, and the reusable assets 306 are stored in the client's buffer 305 controlled by the server 301.

[0029] In the same or another embodiment, when the asset buffer is controlled by the client, the client decides which assets should be in the buffer. In such a case, the client sends a list of all assets in its buffer to the server. The client can also send a list of assets that have been added or removed from the buffer to the server. For example, referring to FIG. 4 showing an example 400 of client-controlled buffering, the server 401 splits a scene 411 into two scenes 407 and 408 with respective assets "scene 1" 409 and "scene 2" 410. The client 402 streams "scene 1" 404 and "scene 2" 403, and the reusable asset 406 is stored in the client's buffer 405 controlled by the client 401.

[0030] In another embodiment, different strategies can be used to strategize the storage of assets. Depth-based or priority-based methods can be used. For both, priority-based storage and deletion can be used. Instead of all assets being deleted at once, the lowest priority assets are deleted first, and so on.

[0031] In another embodiment, asset sharing can occur on a LAN. This is applicable when multiple devices in the LAN are streaming the same media. In this case, different devices in the network can cooperate with each other to share different assets with each other.

[0032] In the same or another embodiment, assets may also be stored in the buffer based on reusability priority. Thus, a priority is assigned to every asset based on how many times the asset appears in the respective scene. The asset that appears most in such a scene may be assigned the highest priority, and the asset that appears least may be assigned the lowest priority. Asset priority may also be based on the size of the asset. More complex assets may be given a higher priority compared to less complex assets.

[0033] The techniques for asset reusability of light field and / or holographic media described above can be implemented as computer software using computer readable instructions and can be physically stored on one or more computer readable media. For example, Figure 5 illustrates a computer system 500 suitable for implementing certain embodiments of the disclosed subject matter.

[0034] Computer software can be encoded using any suitable machine code or computer language that can be subjected to mechanisms such as assembly, compilation, linking, etc. to create code containing instructions that can be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc. directly, or via interpretation, microcode execution, etc.

[0035] The instructions may be executed on various types of computers or computer components including, for example, personal computers, tablet computers, servers, smart phones, gaming consoles, Internet of Things devices, and the like.

[0036] 5 for computer system 500 are exemplary and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system 500.

[0037] The computer system 500 may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, via tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not shown). Human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (e.g., voice, music, ambient sounds, etc.), images (e.g., scanned images, photographic images obtained from a still camera, etc.), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0038] The input human interface devices may include one or more of a keyboard 501, a mouse 502, a trackpad 503, a touch screen 510, a data glove (not shown), a joystick 505, a microphone 506, a scanner 507, and a camera 508 (only one of each is shown).

[0039] The computer system 500 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen 510, data gloves (not shown), or joystick 505, although there may be haptic feedback devices that do not function as input devices), audio output devices (such as speakers 509, headphones (not shown)), visual output devices (such as screens 510, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without haptic feedback capabilities, some of which may be capable of outputting two-dimensional visual output or three-dimensional or higher output via means such as stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0040] The computer system 500 may also include human-accessible storage devices and their associated media, such as optical media, including a CD / DVD ROM / RW Z20 with CD / DVD or similar media 521, thumb drives 522, removable hard drives or solid state drives 523, legacy magnetic media such as tapes and floppy disks (not depicted), and specialized ROM / ASIC / PLD based devices (not depicted) such as security dongles.

[0041] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0042] The computer system 500 may also include interfaces to one or more communication networks. The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable, satellite and terrestrial television, vehicular and industrial including CANBus, etc. Certain networks generally require an external network interface adapter connected to a particular general-purpose data port or peripheral bus (549) (e.g., a USB port on the computer system 500, etc.), while others are generally integrated into the core of the computer system 500 by connection to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system 500 may communicate with other entities. Such communications may be, for example, unidirectional, receive only (e.g., broadcast television), unidirectional transmit only (e.g., CANbus to a particular CANbus device), or bidirectional, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used with each of those networks and network interfaces described above.

[0043] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to the core 540 of the computer system 500.

[0044] The core 540 may include one or more central processing units (CPUs) 541, graphics processing units (GPUs) 542, dedicated programmable processing units in the form of field programmable gate areas (FPGAs) 543, and hardware accelerators 544 for specific tasks, etc. These devices may be connected via a system bus 548, along with read-only memory (ROM) 545, random access memory 546, and internal mass storage devices 547, such as internal hard drives or SSDs, that are not accessible to the user. In some computer systems, the system bus 548 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs and GPUs, etc. Peripheral devices may be attached directly to the core's system bus 548 or via a peripheral bus 549. Architectures for peripheral buses include PCI, USB, etc.

[0045] The CPU 541, GPU 542, FPGA 543 and accelerator 544 may combine to execute certain instructions that may constitute the aforementioned computer code. That computer code may be stored in ROM 545 or RAM 546. Persistent data may be stored, for example, in internal mass storage device 547, while transitory data may also be stored in RAM 546. Rapid storage and retrieval from any of the memory devices may be enabled using cache memories that may be closely associated with one or more of the CPU 541, GPU 542, mass storage device 547, ROM 545, RAM 546, etc.

[0046] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0047] By way of example and not by way of limitation, computer system 500 having the architecture, and specifically core 540, may provide functionality as a result of processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage devices as described above, as well as media associated with specific storage devices of core 540 that are non-transitory in nature, such as core-internal mass storage device 547 or ROM 545. Software implementing various embodiments of the present disclosure may be stored in such devices and executed by core 540. Computer-readable media may include one or more memory devices or chips, depending on the particular needs. The software may cause core 540, and specifically the processors therein (including CPUs, GPUs, FPGAs, etc.) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM 546 and modifying such data structures according to the processes defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 544) that may operate in place of or in conjunction with software to perform certain processes, or certain portions of certain processes, described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that store software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0048] While this disclosure describes several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are thus within its spirit and scope. [Explanation of symbols]

[0049] 100 example, 101 object, 102 object, 103 object, 104 camera, 105 distance, 106 distance, 107 distance, 200 streaming, 201 asset, 202 asset, 203 asset, 204 camera, 300 example, 301 server, 302 client, 303 scene 2, 304 scene 1, 305 buffer, 306 reusable asset, 307 scene, 308 scene, 309 scene 1, 310 scene 2, 311 scene, 400 example, 401 server, 402 client, 403 scene 2, 404 scene 1, 405 buffer, 406 reusable asset, 407 scene, 408 scene, 409 asset "scene 1", 410 asset "scene 2", 411 scene, 500 computer system, 501 keyboard, 502 mouse, 503 trackpad, 505 joystick, 506 microphone, 507 scanner, 508 camera, 510 touch screen, 521 media, 522 thumb drive, 523 solid state drive, 540 core, 541 central processing unit (CPU), 542 graphics processing unit (GPU), 543 field programmable gate area (FPGA), 544 accelerator, 545 read only memory (ROM), 546 random access memory (RAM), 547 internal mass storage device, 548 system bus, 549 peripheral bus, 600 examples

Claims

1. 1. A method implemented by at least one hardware processor, comprising: controlling a light field or holographic media scene that is streamed to a client; determining a bandwidth available to the client for streaming the light field or holographic media based at least on a state of a client buffer associated with the client; Dividing the light field or holographic media scene into a number of split scenes, each split scene including one or more assets of the scene; determining whether the one or more assets are present in the client buffer; if it is determined that the one or more assets are present in the client buffer, mapping the one or more assets of the scene determined to be present in the client buffer to one or more assets of the scene included in the respective split scene; controlling transmission of a subset of the one or more assets to the client buffer based on the bandwidth, the mapping, and whether the one or more of the assets are determined to be present in the client buffer; and controlling the client to reuse at least one of the assets when rendering at least one of the split scenes.

2. 2. The method of claim 1, wherein the at least one hardware processor is in a server in network communication with the client.

3. tracking which of the assets are sent to the client for each of the preceding and succeeding scenes of the split scenes; The method of claim 2 , further comprising:

4. The method of claim 1 , wherein the at least one hardware processor is of the client.

5. 2. The method of claim 1, wherein controlling the transmission of one or more assets of the plurality of assets to the client buffer is further based on a relative depth of the assets within the scene of the light field or holographic media.

6. 2. The method of claim 1, wherein controlling the transmission of the one or more assets to the client buffer is further based on a relative priority of the assets within the scene of the light field or holographic media.

7. 2. The method of claim 1 , wherein determining whether the one or more of the assets are present in the client buffer further comprises determining whether the one or more of the assets are present in any of a plurality of client buffers, the client buffers each including a client buffer that is networked over a local area network (LAN).

8. The method of claim 1 , further comprising: assigning a reusability priority to the at least one of the assets based on at least one of the number of times the at least one of the assets appears in the scene.

9. 9. The method of claim 8, wherein controlling the client to reuse the at least one of the assets when rendering the at least one of the split scenes is based on the reusability priority assigned to the at least one of the assets.

10. The method of claim 9 , wherein assigning the reusability priority to the at least one of the assets is further based on at least one of a size of the at least one of the assets and a complexity of the at least one of the assets.

11. at least one memory configured to store computer program code; 11. Apparatus comprising at least one hardware processor configured to access said computer program code and to operate as instructed by said computer program code, said computer program code causing said at least one hardware processor to perform a method according to any one of claims 1 to 10.

12. A program for causing a computer to execute processes, the processes causing the computer to execute a method according to any one of claims 1 to 10.

Images