System and method for multiplexed rendering of light fields

Abstract

An exemplary method according to some embodiments may include the following: receiving a media manifest file identifying a plurality of representations of a multiview video—at least one of the plurality of representations having a first subsampling of the view and at least one of the plurality of representations having a second subsampling of the view different from the first subsampling of the view—selecting a selected representation from the plurality of representations; retrieving a selected representation; and rendering the selected representation.

Description

Technology Field

[0001] This application is the formal application of U.S. Provisional Application No. 62 / 823,714, filed on March 26, 2019, titled “System and Method for Multiplexed Rendering of Light Fields,” and claims the benefit therefrom under 35 USC §119(e), the entirety of which is incorporated herein by reference. Background Technology

[0002] Many display technologies are emerging that enable adjustable viewpoint viewing of video content. The light field used by such adjustable viewpoint displays may contain a vast amount of data. Handling light field data can become a bottleneck for the network. [Prior Art][Patent] Patent Publication No. 10-2016-0065742 (Publication Date: June 9, 2016)

[0003] An exemplary method according to some embodiments may include the following: receiving a media manifest file identifying a plurality of representations of a multi-view video—at least one of the plurality of representations having a first subsampling of the view and at least one of the plurality of representations having a second subsampling of the view different from the first subsampling of the view—selecting a selected representation from the plurality of representations; retrieving a selected representation; and rendering a selected representation.

[0004] In some embodiments of the exemplary method, each of the multiple expressions may have a density of views facing each specific direction.

[0005] In some embodiments of the exemplary method, the media manifest file may identify each density of the view and each specific orientation for one or more representations of a plurality of representations.

[0006] In some embodiments of the exemplary method, two or more different subsamplings of a view may differ, at least, in relation to the density of the view of the subsampling of the view facing a particular direction.

[0007] Some embodiments of the exemplary method may further include tracking the user's view direction, wherein selecting the selected representation may include selecting one subsampling of the view among two or more subsamplings of the view, and the selected subsampling of the view has a high density view facing the user's tracked view direction.

[0008] Some embodiments of the exemplary method may further include tracking the user's view direction, wherein selecting a selected representation may include selecting a selected representation based on the user's tracked view direction.

[0009] Some embodiments of the exemplary method may further include obtaining the user's view direction, wherein selecting a selected representation includes selecting a selected representation based on the user's obtained view direction.

[0010] In some embodiments of the exemplary method, the selection of the chosen expression may be based on the user's position.

[0011] In some embodiments of the exemplary method, the selection of the chosen expression may be based on bandwidth constraints.

[0012] In some embodiments of the exemplary method, at least one of the plurality of representations may include a view of higher density with respect to the first view direction than to the second view direction.

[0013] Some embodiments of the exemplary method may further include generating a signal for display using a rendered representation.

[0014] Some embodiments of the exemplary method may further include tracking the user's head position, and the selection of the selected expression may be based on the user's head position.

[0015] Some embodiments of the exemplary method may further include tracking the direction of the user's gaze, and the selection of the selected expression may be based on the direction of the user's gaze.

[0016] Some embodiments of the exemplary method may further include determining the user's viewpoint using the direction of the user's gaze, wherein selecting the selected expression may include selecting the selected expression based on the user's viewpoint.

[0017] Some embodiments of the exemplary method may further include determining the user's viewpoint using the direction of the user's gaze; and selecting at least one subsampling of the view of the multiview video, wherein selecting at least one subsampling of the view may include selecting at least one subsampling of the view within a threshold viewpoint angle of the user's viewpoint.

[0018] Some embodiments of the exemplary method may further include interpolating at least one view, wherein the selected representation may be selected from a plurality of representations and at least one view.

[0019] In some embodiments of the exemplary method, the media manifest file may contain priority data for one or more views, and interpolating at least one view uses the priority data.

[0020] In some embodiments of the exemplary method, the media manifest file may contain priority data for one or more views, and selecting the selected representation uses the priority data.

[0021] Some embodiments of the exemplary method may further include: obtaining light field content associated with a selected representation; decoding a frame of light field content; combining one or more views represented in the frame to generate a combined view composite result; and rendering the combined view composite result to a display.

[0022] In some embodiments of the exemplary method, the frame may include a frame-packed representation of two or more views corresponding to a selected representation.

[0023] In some embodiments of the exemplary method, for at least one of the plurality of representations, the media manifest file may include information corresponding to two or more views of the light field content.

[0024] In some embodiments of the exemplary method, the selection of the selected representation may be based on at least one of the following criteria: user head position, eye tracking, complexity of the content, display capability, and bandwidth available for retrieving light field content.

[0025] In some embodiments of the exemplary method, selecting a selected expression may include: predicting the user's point in time; and selecting a selected expression based on the user's predicted point in time.

[0026] Some embodiments of the exemplary method may further include the following: obtaining light field content associated with a selected representation; generating a generated view of the light field content from the obtained light field content; and rendering the generated view to a display.

[0027] In some embodiments of the exemplary method, generating a generated view of light field content may include: decoding a frame of light field content; and generating a generated view by combining two or more views represented in the frame, wherein the light field content obtained may include a frame of light field content.

[0028] Some embodiments of the exemplary method may further include: decoding a frame of light field content; and combining two or more views represented in the frame to generate a combined view composite result, wherein a plurality of representations of the multiview video may include a plurality of subsamplings of the views of the light field content, and rendering the selected representation may include rendering the combined view composite result to a display.

[0029] Some embodiments of the exemplary method may further include: requesting a media manifest file from a server; and requesting light field content associated with a selected view subset, wherein obtaining light field content associated with a selected view subset may include executing a process selected from the group consisting of: retrieving light field content associated with a selected view subset from a server; requesting light field content associated with a selected view subset from a server; and receiving light field content associated with a selected view subset.

[0030] In some embodiments of the exemplary method, combining two or more views represented in a frame may include using view composition techniques.

[0031] In some embodiments of the exemplary method, selecting one of a plurality of subsamplings of views may also select a view subset based on at least one of the following criteria: user head position, eye tracking, complexity of content, display capability, and bandwidth available for retrieving light field content.

[0032] In some embodiments of the exemplary method, selecting one of a plurality of subsamplings of a view may include: predicting the user's viewpoint; and selecting a subset of views based on the user's predicted viewpoint.

[0033] An exemplary device according to some embodiments may include: a processor; and a non-transient computer-readable medium storing instructions that, when executed by the processor, operate to perform any one of the methods listed above.

[0034] An exemplary method according to some embodiments may include the following: receiving a media manifest file containing information about a plurality of subsampled representations of a view of light field video content; selecting one of the plurality of subsampled representations; acquiring the selected subsampled representation; interpolating one or more sub-views from the selected subsampled representation using information within the manifest file corresponding to each of the one or more views; compositing one or more composite views from one or more sub-views; and displaying one or more composite views.

[0035] Some embodiments of the exemplary method may further include estimating the bandwidth available for streaming light field video content, and as a result, selecting a subsampled representation among a plurality of subsampled representations is based on the estimated bandwidth.

[0036] Some embodiments of the exemplary method may further include tracking the user's position, and as a result, selecting a subsampled representation among a plurality of subsampled representations is based on the user's position.

[0037] Some embodiments of the exemplary method may further include requesting light field video content from a server, wherein obtaining a selected subsampled representation may include executing a process selected from the group consisting of: retrieving a selected subsampled representation from a server, requesting a selected subsampled representation from a server, and receiving a selected subsampled representation.

[0038] In some embodiments of the exemplary method, the information within the manifest file may include location data for two or more views.

[0039] In some embodiments of the exemplary method, the information in the manifest file may include interpolation priority data for one or more of the plurality of views, and the selection of one of the plurality of subsampled representations may be based on the interpolation priority data for one or more of the plurality of views.

[0040] Some embodiments of the exemplary method may further include tracking the user's head position, wherein the selection of one of a plurality of subsampled representations is based on the user's head position.

[0041] Some embodiments of the exemplary method may further include tracking the direction of the user's gaze, wherein the selection of one of a plurality of subsampled representations may be based on the direction of the user's gaze.

[0042] Some embodiments of the exemplary method may further include: determining the user's viewpoint from the direction of the user's gaze; and selecting one or more subviews of light field video content from a group comprising one or more interpolated views and selected subsampled representations, wherein compositing one or more composite views from one or more interpolated subviews may include compositing one or more composite views of the light field using one or more selected subviews and the user's viewpoint.

[0043] Some embodiments of the exemplary method may further include displaying a composite view of the light field.

[0044] In some embodiments of the exemplary method, selecting one or more subviews of the light field may include selecting one or more subviews within a threshold angle of the user's viewpoint.

[0045] Some embodiments of the exemplary method may further include determining the user's viewpoint from the direction of the user's gaze, wherein selecting one of a plurality of subsampled representations may include selecting a subsampled representation based on the user's viewpoint.

[0046] An exemplary device according to some embodiments may include: a processor; and a non-transient computer-readable medium storing instructions that, when executed by the processor, operate to perform any one of the methods listed above.

[0047] An exemplary method according to some embodiments may include: receiving a media manifest file containing information on a plurality of subsamplings of a view of light field content; selecting one of the plurality of subsamplings of the view; acquiring light field content associated with the selected subsampling of the view; decoding a frame of light field content; combining two or more views represented in the frame to generate a combined view composite result; and rendering the combined view composite result to a display, wherein the acquired light field content includes a frame of light field content.

[0048] Some embodiments of the exemplary method may further include: requesting a media manifest file from a server; and requesting light field content associated with a selected view subset, wherein obtaining light field content associated with a selected view subset may include executing a process selected from the group consisting of: retrieving light field content associated with a selected view subset from a server; requesting light field content associated with a selected view subset from a server; and receiving light field content associated with a selected view subset.

[0049] In some embodiments of the exemplary method, the frame may include a frame-packed representation of two or more views corresponding to a selected view subset.

[0050] In some embodiments of the exemplary method, the media manifest file may contain information corresponding to two or more views corresponding to a selected view subset.

[0051] In some embodiments of the exemplary method, for at least one of a plurality of subsamplings of a view, the media manifest file may include information corresponding to two or more views of light field content.

[0052] In some embodiments of the exemplary method, selecting one of a plurality of subsamplings of a view may include parsing information within a media manifest file for a plurality of subsamplings of a view of light field content.

[0053] In some embodiments of the exemplary method, combining two or more views represented in a frame may include using view composition techniques.

[0054] In some embodiments of the exemplary method, selecting one of a plurality of subsamplings of views may also select a view subset based on at least one of the following criteria: eye tracking, complexity of the content, display capability, and bandwidth available for retrieving light field content.

[0055] In some embodiments of the exemplary method, selecting one of a plurality of subsamplings of a view may include: predicting the user's viewpoint; and selecting a subset of views based on the user's predicted viewpoint.

[0056] An exemplary device according to some embodiments may include: a processor; and a non-transient computer-readable medium storing instructions that, when executed by the processor, operate to perform any one of the methods listed above.

[0057] An exemplary method according to some embodiments may include the following: receiving a media manifest file containing information on a plurality of subsamplings of a view of light field content; selecting one of the plurality of subsamplings of the view; obtaining light field content associated with the selected view subset; generating a view from the obtained light field content; and rendering the generated view to a display.

[0058] In some embodiments of the exemplary method, generating a view from the acquired light field content may include generating a generated view by interpolating the view from the light field content associated with the selected view subset using information within the manifest file corresponding to each view.

[0059] In some embodiments of the exemplary method, generating one or more views from the acquired light field content may include: decoding a frame of the light field content; and generating a generated view by combining two or more views represented in the frame, wherein the acquired light field content may include a frame of the light field content.

[0060] An exemplary device according to some embodiments may include: a processor; and a non-transient computer-readable medium storing instructions that, when executed by the processor, operate to perform any one of the methods listed above.

[0061] An exemplary method according to some embodiments may include the following: receiving a media manifest file identifying multiple subsamplings of a view of a multiview video—wherein multiple subsamplings of a view include two or more different densities of a view—selecting a selected subsampling from the multiple subsamplings of a view; retrieving the selected subsampling; and rendering the selected subsampling.

[0062] An exemplary device according to some embodiments may include: a processor; and a non-transient computer-readable medium storing instructions that, when executed by the processor, operate to perform any one of the methods listed above.

[0063] An exemplary method according to some embodiments may include: rendering a representation of a view containing an entire array of light field video content; transmitting the rendered entire array representation of the view; obtaining the client's current viewpoint; predicting a future viewpoint using the current viewpoint and a viewpoint motion model; prioritizing a plurality of subsampled representations of the view of light field video content; rendering a plurality of prioritized subsampled representations of the view of light field video content; and transmitting a plurality of prioritized subsampled representations of the view.

[0064] An exemplary device according to some embodiments may include: a processor; and a non-transient computer-readable medium storing instructions that, when executed by the processor, operate to perform any one of the methods listed above.

[0065] An exemplary method according to some embodiments may include the following: selecting a plurality of sub-views of light field video content; generating streaming data for each of the plurality of sub-views of light field video content; and generating a media manifest tile containing streaming data for each of the plurality of sub-views of light field video content.

[0066] An exemplary device according to some embodiments may include: a processor; and a non-transient computer-readable medium storing instructions that, when executed by the processor, operate to perform any one of the methods listed above.

[0067] An exemplary method according to some embodiments may include the following: receiving a request for information about light field video content; transmitting a media manifest file containing information about a plurality of subsampled representations of a view of light field video content if the request is a new session request; and transmitting a data segment containing a subset of light field video content if the request is a subset data segment request.

[0068] An exemplary device according to some embodiments may include: a processor; and a non-transient computer-readable medium storing instructions that, when executed by the processor, operate to perform any one of the methods listed above.

[0069] An exemplary signal according to some embodiments may include a signal conveying a representation of a view comprising a whole array of light field video content and a plurality of subsampled representations of a view of the light field video content.

[0070] An exemplary signal according to some embodiments may include a signal that conveys a plurality of sub-views of light field video content.

[0071] An exemplary signal according to some embodiments may include a signal that transmits streaming data for each of a plurality of sub-views of light field video content.

[0072] An exemplary signal according to some embodiments may include a signal that conveys information about a plurality of subsampled representations of a view of light field video content.

[0073] An exemplary signal according to some embodiments may include a signal that conveys a data segment containing a subset of light field video content. Brief explanation of the drawing

[0074] FIG. 1a is a system diagram of an exemplary system illustrating an exemplary communication system according to some embodiments. FIG. 1b is a system diagram of an exemplary system illustrating an exemplary wireless transmit / receive unit (WTRU) that may be used within the communication system illustrated in FIG. 1a according to some embodiments. FIG. 2 is a schematic example illustrating an exemplary full light field rendered with a 5×5 subview according to some embodiment. FIG. 3 is a schematic example illustrating exemplary subsampling using a 3×3 sub-view according to some embodiments. FIG. 4 is a schematic example illustrating an exemplary 5×5 subview assigned priority based on an estimated time according to some embodiments. FIG. 5 is a schematic example illustrating an exemplary adjusted sub-view priority based on an updated time-view motion model according to some embodiments. FIG. 6a is a diagram illustrating an exemplary full-image 5×5 array light field configuration corresponding to FIG. 2 according to some embodiments. FIG. 6b is a diagram illustrating an exemplary uniform subsampled 3×3 array light field configuration corresponding to FIG. 3 according to some embodiment. FIG. 6c is a diagram illustrating an exemplary center-biased and subsampled 3×3 array light field configuration corresponding to FIG. 4 according to some embodiments. FIG. 6d is a diagram illustrating an exemplary left-biased and subsampled 3×3 array light field configuration corresponding to FIG. 5 according to some embodiment. FIG. 7 is a schematic example illustrating an exemplary numbering configuration of a light field sub-view according to some embodiments. FIG. 8 is a diagram illustrating an exemplary MPEG-DASH Media Presentation Description (MPD) file according to some embodiments. FIG. 9 is a diagram illustrating an exemplary MPD file having light field configuration elements according to some embodiments. FIG. 10 is a system diagram illustrating an exemplary set of interfaces for multiplexed light field rendering according to some embodiments. FIG. 11 is a message sequencing diagram illustrating an exemplary process for multiplexed light field rendering using an exemplary client pull model according to some embodiments. FIG. 12 is a message sequencing diagram illustrating an exemplary process for multiplexed light field rendering using an exemplary server push model according to some embodiments. FIG. 13 is a message sequencing diagram illustrating an exemplary process for multiplexed light field rendering using an exemplary client pool model according to some embodiments. FIG. 14 is a flowchart illustrating an exemplary process for a content server according to some embodiments. FIG. 15 is a flowchart illustrating an exemplary process for a viewing client according to some embodiments. FIG. 16 is a schematic example illustrating an exemplary light field rendered with a 5×5 subview according to some embodiment. FIG. 17 is a schematic example illustrating an exemplary light field rendered with a 3×3 subview according to some embodiment. FIG. 18 is a schematic example illustrating an exemplary light field rendered with a 2×3 subview according to some embodiment. FIG. 19a is a diagram illustrating an exemplary 5×5 array light field configuration corresponding to FIG. 16 according to some embodiment. FIG. 19b is a diagram illustrating an exemplary 3×3 array light field configuration corresponding to FIG. 17 according to some embodiments. FIG. 19c is a diagram illustrating an exemplary 2×3 array light field configuration corresponding to FIG. 18 according to some embodiments. FIG. 20 is a schematic example illustrating exemplary subsampling using a 3×3 sub-view according to some embodiments. FIG. 21 is a schematic example illustrating an exemplary numbering configuration of a light field sub-view according to some embodiments. FIG. 22 is a diagram illustrating an exemplary MPEG-DASH media presentation description (MPD) file according to some embodiments. FIG. 23 is a diagram illustrating an exemplary MPD file having light field configuration elements according to some embodiments. FIG. 24 is a system diagram illustrating an exemplary set of interfaces for adaptive light field streaming according to some embodiments. FIG. 25 is a message sequencing diagram illustrating an exemplary process for adaptive light field streaming using estimated bandwidth and view interpolation according to some embodiments. FIG. 26 is a message sequencing diagram illustrating an exemplary process for adaptive light field streaming using predicted view positions according to some embodiments. FIG. 27 is a flowchart illustrating an exemplary process for content server preprocessing according to some embodiments. FIG. 28 is a flowchart illustrating an exemplary process for content server runtime processing according to some embodiments. FIG. 29 is a flowchart illustrating an exemplary process for a viewing client according to some embodiments. FIG. 30 is a flowchart illustrating an exemplary process for a viewing client according to some embodiments. FIG. 31 is a flowchart illustrating an exemplary process for a viewing client according to some embodiments. FIG. 32 is a flowchart illustrating an exemplary process for a viewing client according to some embodiments. Entities, connections, arrangements, and the like, which are depicted in various drawings—and described in relation thereto—are presented as examples rather than as limitations. As such, any and all statements or other indications regarding what a particular drawing “depicts,” what a particular element or entity of a particular drawing “is” or “has”, and any and all statements—which may be read as absolute and therefore limiting when separated and out of context—may be appropriately read only as structurally preceded by a clause such as “in at least one embodiment, …”. For the sake of simplification and clarification of expression, this implied precedence clause is not repeated tediously in the detailed description. Specific details for implementing the invention

[0075] An exemplary network for implementing an embodiment

[0076] The wireless transceiver unit (WTRU) may be used as a viewing client, for example, in some embodiments described herein.

[0077] FIG. 1a is a drawing illustrating an exemplary communication system (100) in which one or more disclosed embodiments may be implemented. The communication system (100) may be a multiple access system that provides content such as voice, data, video, messaging, broadcast, etc. to multiple wireless users. The communication system (100) may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, a communication system (100) may utilize one or more channel access methods such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block-filtered OFDM, filter bank multicarrier (FBMC), and others.

[0078] As illustrated in FIG. 1a, the communication system (100) may include wireless transceiver units (WTRUs) (102a, 102b, 102c, 102d), a RAN (104 / 113), a CN (106), a public switched telephone network (PSTN) (108), the Internet (110), and other networks (112), but it will be recognized that the disclosed embodiment considers any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs (102a, 102b, 102c, 102d) may be any type of device configured to operate and / or communicate in a wireless environment. For example, WTRUs (102a, 102b, 102c, 102d) may include any of which may be referred to as a “station” and / or “STA”, configured to transmit and / or receive wireless signals, user equipment (UE), mobile station, fixed or mobile subscriber unit, subscription-based unit, pager, cellular phone, personal digital assistant (PDA), smartphone, laptop, netbook, personal computer, wireless sensor, hotspot or Mi-Fi device, Internet of Things (IoT) device, watch or other wearable, head-mounted display (HMD), vehicle, drone, medical device and application (e.g., remote surgery), industrial device and application (e.g., robot and / or other wireless device operating in the context of an industrial and / or automated processing chain), consumer electronics device, device operating on a commercial and / or industrial wireless network, and the like. Any of WTRU(102a, 102b, 102c and 102d) may be interchangeably referred to as UE.

[0079] The communication system (100) may also include a base station (114a) and / or a base station (114b). Each of the base stations (114a, 114b) may be any type of device configured to wirelessly interface with at least one of the WTRUs (102a, 102b, 102c, 102d) to facilitate access to one or more communication networks such as CN (106), the Internet (110), and / or other networks (112). For example, the base stations (114a, 114b) may be a base transceiver station (BTS), Node-B, eNode B, Home Node B, Home eNode B, gNB, NR Node B, site controller, access point (AP), wireless router, and the like. Although each base station (114a, 114b) is described as a single element, it will be recognized that the base station (114a, 114b) may include any number of interconnected base station and / or network elements.

[0080] A base station (114a) may be part of a RAN (104 / 113) which may also include other base station and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), a relay node, etc. Base station (114a) and / or base station (114b) may be configured to transmit and / or receive radio signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage for radio services in a specific geographical area that may be relatively fixed or may vary over time. A cell may be further divided into cell sectors. For example, a cell associated with a base station (114a) may be divided into three sectors. Accordingly, in one embodiment, the base station (114a) may include three transceivers, that is, one transceiver for each sector of the cell. In one embodiment, the base station (114a) may utilize multiple-input multiple output (MIMO) technology and thus may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.

[0081] Base stations (114a, 114b) may communicate with one or more of WTRUs (102a, 102b, 102c, 102d) via an air interface (116), which may be any suitable radio communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface (116) may be established using any suitable radio access technology (RAT).

[0082] More specifically, as mentioned above, the communication system (100) may be a multiple access system and may utilize one or more channel access methods such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and so on. For example, base stations (114a) and WTRUs (102a, 102b, 102c) within a RAN (104 / 113) may implement radio technologies such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish a radio interface (116) using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High-Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High-Speed ​​UL Packet Access (HSUPA).

[0083] In one embodiment, the base station (114a) and WTRU (102a, 102b, 102c) may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA) which may establish a radio interface (116) using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro).

[0084] In one embodiment, the base station (114a) and WTRU (102a, 102b, 102c) may implement wireless technology such as NR wireless access, which may establish a wireless interface (116) using new radio (NR).

[0085] In one embodiment, the base station (114a) and the WTRU (102a, 102b, 102c) may implement multiple wireless access technologies. For example, the base station (114a) and the WTRU (102a, 102b, 102c) may implement LTE wireless access and NR wireless access together, for example, using the dual connectivity (DC) principle. Accordingly, the wireless interface utilized by the WTRU (102a, 102b, 102c) may be characterized by transmissions and / or multiple types of wireless access technologies transmitted to / from multiple types of base stations (e.g., eNB and gNB).

[0086] In another embodiment, the base station (114a) and WTRU (102a, 102b, 102c) may implement wireless technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi)), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, IS-2000 (Interim Standard 2000), IS-95 (Interim Standard 95), IS-856 (Interim Standard 856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and others.

[0087] The base station (114b) of FIG. 1a may be, for example, a wireless router, home node B, home eNode B, or access point, and may utilize any suitable RAT to facilitate wireless connectivity in localized areas such as workplaces, homes, vehicles, campuses, industrial facilities, air corridors (e.g., used by drones), roadways, and so on. In one embodiment, the base station (114b) and WTRU (102c, 102d) may implement wireless technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In another embodiment, the base station (114b) and WTRU (102c, 102d) may implement wireless technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). Still in other embodiments, the base station (114b) and WTRU (102c, 102d) may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish a picocell or femtocell. As illustrated in FIG. 1a, the base station (114b) may have a direct connection to the Internet (110). Thus, the base station (114b) may not need to access the Internet (110) via the CN (106).

[0088] The RAN (104 / 113) may communicate with a CN (106), which may be any type of network configured to provide voice, data, application, and / or voice over internet protocol (VoIP) services, including one or more of the WTRUs (102a, 102b, 102c, 102d). The data may have various Quality of Service (QoS) requirements, such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, and so on. The CN (106) may provide call control, billing services, mobile location-based services, prepaid calling, Internet connectivity, video distribution, and so on, and / or perform high-level security functions such as user authentication. Although not illustrated in FIG. 1a, it will be recognized that the RAN (104 / 113) and / or CN (106 / 115) may communicate directly or indirectly with other RANs utilizing the same RAT or different RAT as the RAN (104 / 113). For example, in addition to connecting to the RAN (104 / 113) which may utilize NR wireless technology, the CN (106) may also communicate with other RANs (not illustrated) utilizing GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi wireless technology.

[0089] CN (106) may also function as a gateway for WTRUs (102a, 102b, 102c, 102d) to access the PSTN (108), the Internet (110), and / or other networks (112). The PSTN (108) may include a circuit-switched telephone network providing plain old telephone service (POTS). The Internet (110) may include a global system of interconnected computer networks and devices using common communication protocols such as the transmission control protocol (TCP), user datagram protocol (UDP), and / or internet protocol (IP) in a TCP / IP Internet protocol suite. The network (112) may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, the network (112) may include other CNs connected to one or more RANs that may utilize the same RAT or different RAT as the RAN (104 / 113).

[0090] Some or all of the WTRUs (102a, 102b, 102c, 102d) in the communication system (102d) may include multimode capabilities (for example, the WTRUs (102a, 102b, 102c, 100) may include multiple transceivers for communicating with different wireless networks through different wireless links). For example, the WTRU (102c) shown in FIG. 1a may be configured to communicate with a base station (114a) that may utilize cellular-based wireless technology and a base station (114b) that may utilize IEEE 802 wireless technology.

[0091] FIG. 1b is a system diagram of an exemplary WTRU (102). As illustrated in FIG. 1b, the WTRU (102) may include, among other things, a processor (118), a transceiver (120), a transmit / receive element (122), a speaker / microphone (124), a keypad (126), a display / touchpad (128), non-removable memory (130), removable memory (132), a power supply (134), a global positioning system (GPS) chipset (136), and / or other peripherals (138). It will be recognized that the WTRU (102) may include any noncombination of the aforementioned elements while still conforming to one embodiment.

[0092] The processor (118) may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and the like. The processor (118) may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU (102) to operate in a wireless environment. The processor (118) may be coupled to a transceiver (120) which may be coupled to a transmit / receive element (122). Although FIG. 1b depicts the processor (118) and transceiver (118) as separate components, it will be recognized that the processor (120) and transceiver (120) may be integrated together in an electronic package or chip.

[0093] The transmitting / receiving element (122) may be configured to transmit a signal to a base station (e.g., base station (114a)) or to receive a signal from the base station via the wireless interface (116). For example, in one embodiment, the transmitting / receiving element (122) may be an antenna configured to transmit and / or receive an RF signal. In another embodiment, the transmitting / receiving element (122) may be an emitter / detector configured to transmit and / or receive an IR, UV, or visible light signal, for example. In yet another embodiment, the transmitting / receiving element (122) may be configured to transmit and / or receive both RF and optical signals. It will be recognized that the transmitting / receiving element (122) may be configured to transmit and / or receive any combination of wireless signals.

[0094] Although the transmitting / receiving element (122) is depicted as a single element in FIG. 1b, the WTRU (122) may include any number of transmitting / receiving elements (102). More specifically, the WTRU (102) may utilize MIMO technology. Thus, in one embodiment, the WTRU (102) may include two or more transmitting / receiving elements (122) (e.g., multiple antennas) for transmitting and receiving wireless signals through the wireless interface (116).

[0095] The transceiver (120) may be configured to modulate a signal to be transmitted by the transmitting / receiving element (122) and to demodulate a signal received by the transmitting / receiving element (122). As mentioned above, the WTRU (102) may have multimode capabilities. Accordingly, the transceiver (120) may include multiple transceivers to enable the WTRU (102) to communicate through multiple RATs, such as NR and IEEE 802.11, for example.

[0096] The processor (118) of the WTRU (102) may be coupled to a speaker / microphone (124), a keypad (126), and / or a display / touchpad (128) (e.g., a liquid crystal display (LCD) display unit or an organic light-emitting diode (OLED) display unit) and may receive user input data from these. The processor (118) may also output user data to the speaker / microphone (124), the keypad (126), and / or the display / touchpad (128). Additionally, the processor (118) may access information in any type of suitable memory, such as non-removable memory (130) and / or removable memory (132), and may store data in any type of suitable memory. The non-removable memory (130) may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory (132) may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In another embodiment, the processor (118) may access information in memory that is not physically located on the WTRU (102), such as memory on a server or home computer (not shown), and may store data in that memory.

[0097] The processor (118) may receive power from the power source (134) and may be configured to distribute power to other components of the WTRU (102) and / or control the power. The power source (134) may be any suitable device for supplying power to the WTRU (102). For example, the power source (134) may include one or more dry cell batteries (e.g., nickel cadmium (NiCd), nickel zinc (NiZn), nickel metal hydride (NiMH), lithium ion (Li ion), etc.), solar cells, fuel cells, and the like.

[0098] The processor (118) may also be coupled to a GPS chipset (136) which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU (102). Additionally, in addition to, or instead of, information from the GPS chipset (136), the WTRU (102) may receive location information from a base station (e.g., base station (114a, 114b)) via the wireless interface (116) and / or determine its location based on the timing of signals being received from two or more nearby base stations. It will be recognized that the WTRU (102) may obtain location information through any suitable location determination method while remaining consistent with one embodiment.

[0099] The processor (118) may also be coupled to other peripherals (138) which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals (138) may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photography and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, and the like. The peripherals (138) may include one or more sensors, and the sensors may include a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; Geolocation sensor; may be one or more of an altimeter, light sensor, touch sensor, magnetometer, barometer, gesture sensor, biometric sensor, and / or humidity sensor.

[0100] The WTRU (102) may include a full-duplex radio in which the transmission and reception of some or all of the signal (e.g., associated with a specific subframe for both the UL (e.g., for transmission) and the downlink (e.g., for reception) may be simultaneous and / or concurrent. The full-duplex radio may include an interference management unit for reducing and / or substantially eliminating self-interference through signal processing via a processor (e.g., via a separate processor (not shown) or processor (118)) or hardware (e.g., a choke). In one embodiment, the WRTU (102) may include a half-duplex radio, in which case the transmission and reception of some or all of the signal (e.g., associated with a specific subframe for either the UL (e.g., for transmission) or the downlink (e.g., for reception)

[0101] With regard to FIGS. 1a and 1d, and the corresponding descriptions of FIGS. 1a and 1b, one or more of the functions described herein in connection with one or more of the WTRU (102a-d), base station (114a-d), and / or any other device(s) described herein may be performed by one or more emulation devices (not shown). The emulation device may be one or more devices configured to emulate one or more of the functions described herein, or all of them. For example, the emulation device may be used to test other devices and / or to simulate network and / or WTRU functions.

[0102] Emulation devices may be designed to implement one or more tests of other devices in a lab environment and / or an operator network environment. For example, one or more emulation devices may perform one or all functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices within a communication network. One or more emulation devices may perform one or all functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. Emulation devices may be directly coupled to other devices for testing purposes and / or may perform tests using over-the-air wireless communication.

[0103] One or more emulation devices may perform one or more functions, including the whole, while not being implemented or deployed as part of a wired and / or wireless communication network. For example, the emulation devices may be utilized in test scenarios of a wired and / or wireless communication network that is not deployed (e.g., for testing) and / or in a test laboratory to implement testing of one or more components. One or more emulation devices may be test devices. Wireless communication via direct RF coupling and / or RF circuitry (e.g., which may include one or more antennas) may be used by the emulation devices to transmit and / or receive data.

[0104] details

[0105] A light field generates a vast amount of data that may include a description of the amount of light flowing in a set of directions emitted from a point in space. A high-fidelity light field, as a representation of a 3D scene, may contain a massive amount of data. To support real-time transmission and visualization, efficient data distribution optimization methods may be required, and the amount of light field data rendered and transmitted may be reduced.

[0106] To compress traditional 2D video, various lossless and lossy bitrate reduction and compression methods have been developed. In addition to current spatiotemporal compression methods, one class of bitrate reduction methods integrally transmits portions of information that are multiplexed over time. In CRT displays, multiplexing was widely used as a format for interlacing image lines in analog TV transmission.

[0107] Another class of compression algorithms consists of various prediction methods that may typically be used similarly on both the transmitting side (encoder or server) and the receiving side (decoder). These predictions can be spatial (intra-frame) or temporal (inter-frame). Some of the approaches mentioned above have also been used with light fields. For example, the journal article “Evaluation of the Concept of Dynamic Adaptive Streaming of Light Field Video, IEEE TRANSACTIONS ON BROADCASTING (2018)” by Kara, Peter A., ​​et al. describes how the subjective quality of light field rendering is affected by balancing simple quality switching and stalling (frame freezing) approaches with the associated trade-offs between transmission bitrate, light field angular resolution, and spatial resolution.

[0108] In general, methods applying predictive coding to the real-time transmission of light fields are still rare. An example of light field compression is discussed in the journal article *JPEG Pleno: Toward an Efficient Representation of Visual Reality, 23:4 IEEE MULTIMEDIA 14-20 (Oct.-Dec. 2016)* by Ebrahimi, Touradj, et al. This article describes how existing multiview coding methods (e.g., MPEG HEVC or 3D HEVC) may be used for light field compression. 3D HEVC is an extension of HEVC designed to support depth images.

[0109] H.264 MVC, and subsequent MFPEG HEVC (H.265) and derivatives support several important 3D functionalities. These include viewing content on an external 3D display that projects a set of viewpoints in different angular directions in space. For this purpose, the entire 3D information may generally be transmitted and decoded. The aforementioned standards also support dynamic viewpoints and motion parallax using conventional 2D displays, including HMDs; however, in these applications, receiving a complete set of spatial views for one or several time-varying user viewpoints may not be optimal.

[0110] H.264 MVC, HEVC, and 3D HEVC standards may also be used for coding light field data. The traditional and commonly used format for light fields is the view matrix (integral format), which represents different viewpoints of a scene as a matrix / mosaic of 2D views from adjacent viewpoints. For example, the HEVC standard supports up to 1024 multi-views, which may also be applied to light fields represented by multiple sub-views.

[0111] MPEG video codecs may be used to compress the light fields of multiview formats. In these codecs, the Network Abstraction Layer (NAL) defines high-level data structures, but it also causes limitations in the utilization of sub-view redundancy over time. In particular, according to the journal article “Overview of the Stereo and Multiview Video Coding Extensions of the H.264 / MPEG-4 AVC Standard, 99:4 PROC. IEEE 1-15 (April 2011)” by Vetro, Anthony et al. ("Vetro"), this NAL structure does not allow for the prediction of the subview picture at a given time moment from other subview pictures at different time moments. Furthermore, for reasons of backward compatibility, Vetro is understood to argue that the compressed MVC multiview must include the base view bitstream. In usage scenarios where the base view is not used, or is used only occasionally, this condition leads to excessive bandwidth usage during transmission.

[0112] In existing multiview coding standards, such as those discussed in *Standardized Extensions of High Efficiency Video Coding (HEVC), 7:6 IEEE J. SELECTED TOPICS IN SIGNAL PROC. 1001-16 (Dec. 2013)* by Sullivan, Gary J. et al., it may generally be necessary to decode all views, even when viewing a specific subview. Accordingly, it may not be possible to decode only one subview at a time, except when the subview is an essential primary view.

[0113] The journal article “Client-Driven Selective Streaming of Multiview Video for Interactive 3DTV, 17:11 IEEE TRANS. ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 1558-65 (2007)” by Kurutepe, Engin et al. ("Kurutepe") proposes a modification to the MVC structure to allow individual views of a multiview video to be distributed as an enhancement layer. The enhancement layer is connected to the MVC base layer containing spatially downsampled frames at a low bitrate for all views of the multiview video. As such, the proposed alternative MVC structure is understood to distribute all views, but only as low-quality versions, while, in contrast to a single view or pair of views in the standard MVC, high-resolution versions may be optionally downloaded as enhancement layers. In addition to proposing modifications to the MVC structure, Kurutepe explains the pre-fetching of views based on predicted future user head positions inferred from collected head tracking data.

[0114] Along with the distribution of large data files, the peer-to-peer (P2P) distribution model provides a robust alternative to the strict client-server model, alleviating server connection and bandwidth requirements because clients share a portion of the data they have already downloaded among all other clients. P2P distribution has also been considered for the distribution of visual content. See the article “Adaptive 3D Multi-View Video Streaming Over P2P Networks, CONFERENCE PROCEEDINGS OF 2014 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP) 2462-66 (2014)” by Ozcinar et al. (“Ozcinar”) and , C. Goktug & Tekalp, Murat et al., 『Peer-to-Peer System Design for Adaptive 3D Video Streaming, 51:5 IEEE COMM., 108-114 (2013)』(" ") proposes an adaptive P2P multi-view content distribution solution. In the proposed approach, all clients connected to a streaming session are connected to a mesh network, where each client downloads missing content segments from any client or server within the mesh and simultaneously allows other clients to download segments they have already downloaded. A tracker is used to collect and share information about content segments available on each client. It proposes two variations of a multiview adaptation approach, both of which perform adaptation by asymmetrically degrading the image quality between individual subviews of multiview content, following observations from previous studies on asymmetric image quality performed on stereoscopic views. Ozcinar and Both discuss the distribution of depth information along with color images of multiview content. Ozcinar proposes the distribution of additional metadata as a practical mechanism for handling adaptation on the client side, which pre-selects which multiview image downloads are discarded in the event of network congestion. Experimental results demonstrate that the robustness of P2P multiview streaming using the proposed adaptation method is significantly increased under congestion conditions.

[0115] In addition to a camera array that generates high-resolution sub-views, a lensarray optic that generates light field images in lenslet format may be used for capturing. When using lenslet format for light fields, the content may be converted to a multiview light field before compression (see JPEG Pleno) and then converted back to a lenslet format that the receiver will use.

[0116] The resolution and frame rate of the video sequence may be adapted to enable content streaming via the DASH protocol. Many other devices do not use the angular modulation of light field content in adaptive video streaming. Many other devices also do not change the number of views bundled into a frame-packed multiview representation.

[0117] Multiplexed light field rendering

[0118] Supporting dynamic user viewpoints and motion parallax using many existing multiview coding standards can generally lead to excessive bandwidth usage, particularly in applications that generate only a single view or stereo view 2D image for display at a single time step. Examples of such applications include 3 DoF+ or 6 DoF applications based on viewing content on an HMD or other 2D display.

[0119] To optimize rendering and data distribution, a subset of the entire integral light field data may be generated and transmitted, and the receiver may synthesize additional views from the transmitted views. Furthermore, the impact on perceived image quality resulting from the selection of rendered and transmitted subviews may guide the subview rendering process on either the server-side or the client-side to ensure experience quality.

[0120] In some embodiments, data transmission may be optimized using multiplexed rendering of the light field (which may be on a content server) and view composition (which may be on a viewing client), which may mitigate temporal variations in the light field data caused by multiplexing.

[0121] The viewing client may transmit the current viewpoint to the content server or request a specific light field subview by analyzing the content locally. A viewpoint motion estimation model may be used to predict the client's viewpoint for future time steps. Based on the viewpoint prediction, subviews of the integrated light field may be prioritized. The priority may control the rendering and transmission of individual subviews of the integrated light field.

[0122] In some embodiments, the process of multiplexing light field data may transmit a subset of subviews instead of transmitting all integrated light field images at each time step. Consequently, the content delivery bandwidth used may be reduced. Temporal multiplexing of subviews may be detected in the transmission of data and associated metadata (e.g., streamed subview and virtual camera specifications and timestamps), as well as in the current time point signaling by the client.

[0123] According to some embodiments, server-side processing may be reduced, and content delivery bandwidth requirements may be reduced by dynamically limiting the number of light field subviews rendered and delivered. The content server may continuously render individual light field subviews instead of rendering the entire integral image for each time step in a sequential manner, using content analysis that prioritizes the rendering of subviews to maintain image quality perceived by the viewer on the client side.

[0124] In some embodiments, the viewer uses an HMD or similar mobile device that provides a monocular or stereoscopic view by synthesizing a new view to light field data based on tracking and user input.

[0125] Some embodiments use, for example, an exemplary server push model or an exemplary client pull model for streaming. In the server push model, for example, an exemplary server process may prioritize sub-view rendering and streaming based on the timing signaled by the client, and an exemplary client process may synthesize a time point from light field data when the sub-view of the integrated light field has temporal variability. For example, in a client pull model operating similarly to the MPEG-Dash model, the server may provide the client with a manifest file representing versions of an array image for different viewing positions, and the client may, in some embodiments: determine the number of sub-views based on the viewer position and bandwidth for the collection of sub-views described in the manifest; prioritize the sub-views; and pull data from the content server according to the priority.

[0126] Supporting dynamic viewpoints and motion parallax using existing multiview coding standards can generally lead to excessive bandwidth usage, particularly in applications that require only one viewpoint at a time. Such applications may include, for example, certain 3 DoF+ or 6 DoF applications based on viewing either real-time or stored content on an HMD or other 2D display. The systems and methods described herein according to some embodiments may avoid existing limitations by taking into account the influence of selected, rendered, and transmitted subviews on the current and predicted user viewpoints and the perceived image quality.

[0127] In general, significant savings in bitrate may be achieved by tracking the user and predicting their movements and viewpoints. In “Enabling Multi-party 3D Tele-immersive Environments with ViewCast, 6:2 ACM TRANS. ON MULTIMEDIA COMPUTING, COMM'S, AND APPL'S 111-139 (Mar. 2010)” by Yang, Zhengyu et al. ("Yang"), temporal prediction of the user viewpoint for real-time 3D captured scenes resulted in approximately 70% bitrate savings (when using viewpoint prediction, an average of 2-3 simultaneous captures were coded and transmitted instead of eight 3D captures per scene). Yang provides several examples of user viewpoint prediction techniques. According to some embodiments, the exemplary methods and systems disclosed herein apply user viewpoint prediction, for example, to light field rendering. Yang is not understood to apply user viewpoint prediction to light field rendering.

[0128] Some embodiments reduce the number of subviews rendered and transmitted through content analysis and multiplexed light field rendering. In some embodiments, by analyzing the content based on content features, the estimated user viewpoint location, and the prediction accuracy of the estimated viewpoint location, the content server weights priorities for the entire light field subview, which may ultimately determine the order and frequency in which the subviews are rendered and submitted. Dynamic selection of a set of view subsampling locations may allocate a larger view density near the predicted viewpoint, thereby allowing for enhanced interpolation of views near the predicted viewpoint when the rendered frame reaches the viewing client and is displayed. Coarser view sampling away from the signaled viewpoint may be used to enable data to be displayed for areas within or outside the focus instead of the predicted viewpoint due to inaccuracies in viewpoint motion estimation. FIGS. 2 through 5 illustrate exemplary priorities assigned to light field sub-images.

[0129] FIG. 2 is a schematic example illustrating an exemplary full light field rendered with a 5×5 sub-view according to some embodiment. In FIG. 2, the exemplary full light field (200) is rendered using a 5×5 virtual camera array of sub-views (202). FIG. 2 illustrates an exemplary full light field (200) on which the subsampling examples of FIG. 3 through 5 are based.

[0130] FIG. 3 is a schematic example illustrating exemplary subsampling using 3×3 subviews according to some embodiments. To reduce bandwidth, a 3×3 array of images may be selected from an exemplary entire light field (300) for transmission to a client. In some embodiments, the server may use an estimated viewpoint position predicted by a viewpoint motion model to prioritize the subviews. FIG. 3 illustrates an exemplary 3×3 array of selected subviews (302, 304, 306, 308, 310, 312, 314, 316, 318) in the form of a checkerboard pattern, where the selected subviews are shown as solid black ellipses.

[0131] According to some embodiments, subsampling views may generally reduce transmission bandwidth by using view interpolation used at the receiver to generate additional views. In some embodiments, the location of the subsampled view is specified for the entire grid. Different subsampling priorities may be assigned based on the user's gaze. The priority of individual light field subviews may be determined based on content analysis and context information (e.g., user / gaze location and display capability). Individual sets of subviews may be generated and / or transmitted in a sequential manner based on priority. On the client side, a cache may receive subviews, and the client may use temporally-coherent interpolation of the subviews to synthesize a viewpoint, for example, a new viewpoint.

[0132] FIG. 4 is a schematic example illustrating an exemplary 5×5 subview assigned priority based on an estimated time point according to some embodiments. FIG. 5 is a schematic example illustrating an exemplary adjusted subview priority based on an updated time point motion model according to some embodiments. In some embodiments, the subview (418, 518) closest to the estimated time point is assigned the highest priority (marked by a solid white ellipse) in FIG. 4 and FIG. 5.

[0133] Using the current velocity of the viewpoint movement, the accuracy of the viewpoint location estimation is determined, and subview priority is set. In some embodiments, subview priority may be an indication of how large the range of nearby subviews to be rendered is. In some embodiments, visual differences between previously rendered subviews may be used to weight the priority. Because small shifts in the viewpoint location may cause visual differences (e.g., large depth variations in content or mirror reflections), denser sampling may be used around the estimated most likely viewpoint location, and correspondingly, subview priority may be assigned. In some embodiments, the analysis process may use these metrics and assign the second highest priority to the area surrounding the most likely viewpoint at an appropriate sampling density. In the example shown in FIGS. 4 and 5, the frame assigned the second highest priority is marked with a dashed black ellipse (410, 412, 414, 416, 510, 512, 514, 516).

[0134] According to the example, for a possible point in time, the area outside the focal area (in this example, the area with the highest priority and the second highest priority) is assigned the third highest priority and rendered with a lower sampling density. Consequently, a large portion of the entire light field may be rendered with a lower sampling density. In the example illustrated in FIGS. 4 and 5, the subview assigned the third highest priority is marked by a solid black ellipse (402, 404, 406, 408, 502, 504, 506, 508).

[0135] Figure 5 indicates that the user's viewpoint is estimated to be shifted to the left compared to Figure 4. Sub-views (510, 512, 514, 516) immediately adjacent to the estimated user viewpoint are assigned the second highest priority. Comparing Figures 4 and 5, in response to the shift to the left caused by the estimated user viewpoint, the set of sub-views with the second highest priority (410, 412, 414, 416, 510, 512, 514, 516) also moves to the left.

[0136] In some embodiments, an exemplary rendering process may use assigned priorities to determine how often to render a subview (or, for example, to update the rendering of the subview). For example, the rendering process may send the rendered view to a client along with a timestamp. The timestamp may represent a synchronized time between the viewing client and the virtual camera used by the subview. In some embodiments, a prioritized subset of views (e.g., a 3×3 array) may be used by the light field compression and distribution process or device, along with the original position (or, for example, relative position in some embodiments) of each subview in a larger array (e.g., a 5×5 array).

[0137] In some embodiments, an exemplary viewing client process may include determining the user's viewpoint from the direction of the user's gaze and selecting a subsampled representation of light field content based on the user's viewpoint. In some embodiments, an exemplary viewing client process may include determining the user's viewpoint from the direction of the user's gaze, and as a result, selecting one of a plurality of subsampled representations may include selecting a subsampled representation based on the user's viewpoint. For example, the center point of the subsampled representation may be selected to be within a threshold angle or distance of the user's viewpoint.

[0138] FIG. 6a is a diagram illustrating an exemplary full-image 5×5 array light field configuration corresponding to FIG. 2 according to some embodiments. In some embodiments, a manifest file (e.g., a media manifest file), such as a media presentation description (MPD), may contain details of the entire (or complete) light field array, such as the number of views in an adaptation set, indices for those views, and sampling positions of the views. For example, the entire light field array may have N×M view positions. FIG. 6a illustrates an exemplary naming convention for the entire light field 5×5 array (600). The first number at each coordinate position indicates the row, and the second number indicates the column. Position (1, 1) is at the top-left corner, and position (5, 5) is at the bottom-right corner. Of course, this is merely an example, and, for example, other embodiments may use different naming conventions, such as the example illustrated in FIG. 7.

[0139] FIG. 6b is a diagram illustrating an exemplary uniformly subsampled 3×3 array light field configuration corresponding to FIG. 3 according to some embodiments. The 3×3 array (620) shown in FIG. 6b corresponds to the solid black ellipse in FIG. 3. For example, the uniformly subsampled 3×3 array in FIG. 6b represents a sub-view (5, 3) in the bottom center position. This sub-view corresponds to the black ellipse in FIG. 3 in the 5th (bottom) row and the 3rd (center) column.

[0140] FIG. 6c is a diagram illustrating an exemplary center-biased subsampled 3×3 array light field configuration corresponding to FIG. 4 according to some embodiments. The 3×3 array (640) illustrated in FIG. 6c corresponds to the ellipse in FIG. 4. For example, the center-biased subsampled 3×3 array in FIG. 6c represents a sub-view (3, 4) in the middle-right position. This sub-view corresponds to the black dashed ellipse in FIG. 4 in the third (middle) row and the fourth (center-right) column.

[0141] FIG. 6d is a diagram illustrating an exemplary left-biased subsampled 3×3 array light field configuration corresponding to FIG. 5 according to some embodiment. The 3×3 array (660) illustrated in FIG. 6d corresponds to the ellipse in FIG. 5. For example, the left-biased subsampled 3×3 array in FIG. 6d represents a sub-view (2, 2) in the top center position. This sub-view corresponds to the black dashed ellipse in FIG. 5 in the second (middle top) row and the second (center left) column.

[0142] In some embodiments, the representation may include the entire light field array. In some embodiments, the representation may include a collection of views of the light field array. In some embodiments, the representation may include a subsampling of views of the light field array. In some embodiments, the representation may include a subset of views selected from the entire light field array or a subset of views of the light field array. In some embodiments, the representation (which may be designated as a subsampled representation) may include a subsample of another representation of the light field array. In some embodiments, the subsampling of views may include views corresponding to other directions as well as specific directions, such as the examples illustrated in FIG. 6b, 6c, and 6d, and may include one or more uniform subsamplings. In some embodiments, the collection of views of the light field content may include one or more views having different sampling rates so that the collection of views may correspond to multiple sampling rates.

[0143] FIG. 7 is a schematic example illustrating an exemplary numbering configuration of light field subviews according to some embodiments. The exemplary subview numbering configuration illustrated in FIG. 7 is used in the exemplary MPD illustrated in Table 1 below. FIG. 7 illustrates an exemplary subview numbering configuration for an entire light field 5×5 array (700). The subview numbering configuration of FIG. 7 may be compared with the subview numbering configuration of FIG. 6a. While FIG. 6a illustrates an exemplary coordinate subview numbering configuration, FIG. 7 illustrates an exemplary sequential subview numbering configuration. The top row of the 5×5 array (700) of FIG. 7 numbers subviews 1 through 5 from left to right. The second row numbers subviews 6 through 10 from left to right and continues to the last row, numbering subviews 21 through 25 from left to right. FIGS. 7 through 9 and Table 1 together illustrate exemplary MPDs for an exemplary client pool model.

[0144] FIG. 8 is a diagram illustrating an exemplary MPEG-DASH media presentation description (MPD) file according to some embodiments. An exemplary client pool model (e.g., using MPEG-DASH) may use the standard structure of the MPEG-DASH media presentation description (MPD) (802) illustrated in FIG. 8. The MPD file format (800) illustrated in FIG. 8 may be used to transmit the entire media description downloaded by the viewing client as part of the streaming session initialization. The organizational structure of the MPEG-DASH MPD is illustrated in FIG. 8.

[0145] The top-level duration field (804, 806) may represent a start time and a duration. The MPD (802) may include one or more duration fields (804, 806). The duration field (804, 806) may include one or more adaptation sets (808, 810). The adaptation set (808, 810) may include one or more representation fields (812, 814). Each representation (812, 814) within the adaptation set (808, 810) may contain the same content encoded using different parameters. The representation field (812, 814) may include one or more segments (816, 818). The segment (816, 818) may include one or more sub-segments (824, 826) containing DASH media files. The expression field (812, 814) may be divided into one or more sub-expression fields (820, 822). The sub-expression field (820, 822) may contain information applicable to only one media stream.

[0146] In some embodiments, one or more representation fields of the MPD may include views with a higher density for the first view direction compared to the second view direction. In some embodiments, the media manifest file (e.g., MPD) may include priority data for one or more views, and the process of interpolating views may use the priority data (e.g., to select which views may be interpolated). In some embodiments, the media manifest file (e.g., MPD) may include priority data for one or more views, and the process of selecting representations may use the priority data (e.g., to select a representation with a higher priority). In some embodiments, the media manifest file (e.g., MPD) may include at least one representation having information corresponding to two or more views of light field content. In some embodiments, the information within the manifest file may include location data for two or more views. In some embodiments, information within the manifest file may include interpolation priority data for one or more of a plurality of views, and selecting one of a plurality of subsampled representations is based on interpolation priority data for one or more of the plurality of views. In some embodiments, the process may include tracking the user's view direction, and as a result, selecting a representation selects a representation having a density of views associated with the user's view direction. In some embodiments, the representation may include two or more views having two or more densities of views associated with the user's view direction, such as a density of views associated with the left view direction or a density of views associated with the right view direction.

[0147] FIG. 9 is a diagram illustrating an exemplary MPD file having light field configuration elements according to some embodiments. FIG. 9 illustrates a method in which MPD data (902) enabling an exemplary client pool model of multiplexed light field streaming may be organized according to the MPEG-DASH protocol structure.

[0148] In some embodiments, the MPD structure (900) uses periods (904, 906) as top-level entities. Each period (904, 906) may provide information about a single light field scene. A single scene may be, for example, a continuous light field rendering in which a virtual camera array used for rendering is kept constant. The entire experience may include multiple scenes, each specified in a separate period block. Each period block may include a light field rendering setup that is labeled as a light field description (908) in FIG. 9 and corresponds to a first adaptation set. The light field rendering setup block (908) may include the number of sub-views, the placement of virtual cameras used by the sub-views, and an overview of the scene layout, such as the size of the views and the size and placement of elements in the views. Each period block (904, 906) may include one or more subset blocks (910, 912, 914), each subset block corresponding to an adaptation set. Each subset adaptation set (910, 912, 914) may include one or more different versions of content encoded using different configuration parameters. For example, the content may be encoded using different resolutions, different compression rates, different bitrates, and / or different supported codecs. Each subset (910, 912, 914) may include a set number of subviews dependent on an adaptation set having a resolution according to a representation block, encoded using a specific codec at a specific bitrate for the segment block. Each subset (910, 912, 914) may be divided into short clips contained in subsegment blocks having links to actual video data.

[0149] An adaptation set within the period (904, 906) of the MPD (902) may include a subset (910, 912, 914) of the entire array of views that changes the number of views and the sampling positions of the views within the adaptation set. The adaptation set may include the number of views that exist and the index of the available views. The adaptation set may also indicate the priority level of the subviews. An adaptation set (or, in some embodiments, a subset (910, 912, 914)) may include one or more resolutions (918, 920) each containing one or more bitrates (922, 924, 926). For each resolution (918, 920), there may be a series of time steps 1, 2, ..., N (928, 930, 932). Each time step (928, 930, 932) may have a separate URL (934, 936, 938, 940, 942, 944) for each supported bitrate. In the example illustrated in FIG. 9, there are N resolutions each supporting N bitrates. Additionally, an adaptation set within the periods (904, 906) of the MPD (902) may be used for audio, as illustrated as an audio block (916) in the example of FIG. 9.

[0150] In some embodiments, a manifest file (e.g., a media manifest file or MPD) may be used. The information within the manifest file may include location data for two or more views. In some embodiments, the information within the manifest file may include interpolation priority data for one or more views. In some embodiments, the selection of a subsampled representation may be based on the interpolation priority data for one or more views. For example, the interpolation priority data may be used to determine how often a subview is interpolated or updated.

[0151] Table 1 illustrates a pseudocode version of an exemplary MPD having sub-view configurations for a field as illustrated in Fig. 9 and a 5×5 array light field as illustrated in Fig. 7. The examples shown in Table 1 illustrate three different light field view adaptation sets: full, center, and left. The number of sampled views and the sampled view positions differ between the adaptation sets. For a specific resolution, fewer views result in more pixels per view. Traditional DASH rate and resolution adaptations may also be used within each view category. An example of UHD / HD is shown in the "center" light field view adaptation set.

[0152] period

[0153] o Adaptation Set 1: Light Field Scene Description:

[0154] Sparse view array. Number of 5×5 subviews. Virtual camera position.

[0155] o Adaptation Set 2: Subset 1 (Whole Light Field): Numbers of Sub-views 5×5 (Sub-views 1-25)

[0156] Resolution 1: 1920×1080 pixels per subview

[0157] Bitrate 1: Encoded using Codec XX, Required transmission capacity 35 Mbps

[0158] Bitrate 2: Encoded using Codec XX, Required transmission capacity 31 Mbps

[0159] Resolution 2: 1280×720 pixels per subview

[0160] Bitrate 1: Encoded using Codec XX, Required transmission capacity 22 Mbps

[0161] Bitrate 2: Encoded using Codec XX, Required transmission capacity 19 Mbps

[0162] o Adaptation Set 3: Subset 2 (Coarse Sampling): Numbers of Sub-views 2×2 (Sub-views 1, 5, 21, 25)

[0163] Resolution 1: 1920×1080 pixels per subview

[0164] Bitrate 1: Encoded using Codec XX, Required transmission capacity 9 Mbps

[0165] Bitrate 2: Encoded using Codec XX, Required transmission capacity 8 Mbps

[0166] Resolution 2: 1280×720 pixels per subview

[0167] Bitrate 1: Encoded using Codec XX, Required transmission capacity 6 Mbps

[0168] Bitrate 2: Encoded using Codec XX, Required transmission capacity 5 Mbps

[0169] o Adaptation Set 4: Subset 3 (semi-dense partial sampling): Numbers of sub-views 2×2 (sub-views 2, 6, 8, 12)

[0170] Resolution 1: 1920×1080 pixels per subview

[0171] Bitrate 1: Encoded using Codec XX, Required transmission capacity 9 Mbps

[0172] Bitrate 2: Encoded using Codec XX, Required transmission capacity 8 Mbps

[0173] Resolution 2: 1280×720 pixels per subview

[0174] Bitrate 1: Encoded using Codec XX, Required transmission capacity 6 Mbps

[0175] Bitrate 2: Encoded using Codec XX, Required transmission capacity 5 Mbps

[0176] o Adaptation Set 5: Subset 4 (semi-dense partial sampling): Numbers of sub-views 2×2 (sub-views 4, 8, 10, 14)

[0177] Resolution 1: 1920×1080 pixels per subview

[0178] Bitrate 1: Encoded using Codec XX, Required transmission capacity 9 Mbps

[0179] Bitrate 2: Encoded using Codec XX, Required transmission capacity 8 Mbps

[0180] Resolution 2: 1280×720 pixels per subview

[0181] Bitrate 1: Encoded using Codec XX, Required transmission capacity 6 Mbps

[0182] Bitrate 2: Encoded using Codec XX, Required transmission capacity 5 Mbps

[0183] o Adaptation Set 6: Subset 5 (semi-dense partial sampling): Numbers of sub-views 2×2 (sub-views 12, 16, 18, 22)

[0184] Resolution 1: 1920×1080 pixels per subview

[0185] Bitrate 1: Encoded using Codec XX, Required transmission capacity 9 Mbps

[0186] Bitrate 2: Encoded using Codec XX, Required transmission capacity 8 Mbps

[0187] Resolution 2: 1280×720 pixels per subview

[0188] Bitrate 1: Encoded using Codec XX, Required transmission capacity 6 Mbps

[0189] Bitrate 2: Encoded using Codec XX, Required transmission capacity 5 Mbps

[0190] o Adaptation Set 7: Subset 6 (semi-dense partial sampling): Numbers of sub-views 2×2 (sub-views 14, 18, 20, 24)

[0191] Resolution 1: 1920×1080 pixels per subview

[0192] Bitrate 1: Encoded using Codec XX, Required transmission capacity 9 Mbps

[0193] Bitrate 2: Encoded using Codec XX, Required transmission capacity 8 Mbps

[0194] Resolution 2: 1280×720 pixels per subview

[0195] Bitrate 1: Encoded using Codec XX, Required transmission capacity 6 Mbps

[0196] Bitrate 2: Encoded using Codec XX, Required transmission capacity 5 Mbps

[0197] o Adaptation Set 8: Subset 7 (Single Sub-view): Number of Sub-view 1×1 (Sub-view 1)

[0198] Resolution 1: 1920×1080 pixels per subview

[0199] Bitrate 1: Encoded using Codec XX, Required transmission capacity 3 Mbps

[0200] Bitrate 2: Encoded using Codec XX, Required transmission capacity 2.5 Mbps

[0201] Resolution 2: 1280×720 pixels per subview

[0202] Bitrate 1: Encoded using Codec XX, Required transmission capacity 2 Mbps

[0203] Bitrate 2: Encoded using Codec XX, Required transmission capacity 1.8 Mbps

[0204]

[0205]

[0206]

[0207] o Adaptation Set 32: Subset 31 (Single Sub-view): Number of Sub-view 1×1 (Sub-view 25)

[0208] Resolution 1: 1920×1080 pixels per subview

[0209] Bitrate 1: Encoded using Codec XX, Required transmission capacity 3 Mbps

[0210] Bitrate 2: Encoded using Codec XX, Required transmission capacity 2.5 Mbps

[0211] Resolution 2: 1280×720 pixels per subview

[0212] Bitrate 1: Encoded using Codec XX, Required transmission capacity 2 Mbps

[0213] Bitrate 2: Encoded using Codec XX, Required transmission capacity 1.8 Mbps

[0214] Table 1.

[0215] FIG. 10 is a system diagram illustrating an exemplary set of interfaces for multiplexed light field rendering according to some embodiments. In some embodiments, an exemplary system (1000) having a server (e.g., a content server (1002)) may include a database (1010) of spatial content. The content server (1002) may be able to execute an exemplary process (1012) for analyzing content and performing multiplexed light field rendering. One or more content servers (1002) may be connected to a client (e.g., a viewing client (1004)) via a network (1018). In some embodiments, the viewing client (1004) may include a local cache (1016). The viewing client (1004) may be enabled to execute an exemplary process (1014) for compositing a view (or a sub-view in some embodiments) into the content. A viewing client (1004) may be connected to a display (1006) to be used for displaying content (including a composite view of the content). In some embodiments, the viewing client (1004) may include an input (1008) for receiving tracking data (e.g., user eye tracking data, or, for example, user position data, e.g., user head position data from a headset or head-mounted device (HMD)) and input data. In some embodiments, the viewing client (1004) (or a sensor in some embodiments) may be used to track the direction of the user's gaze. The direction of the user's gaze may be used to select a subsampled representation of the content. For example, the direction of the user's gaze may be used to determine the user's viewpoint (and associated subviews), such as the exemplary subview indicated by the white ellipse in FIGS. 4 and 5.In some embodiments, to determine the sub-view closest to the user's gaze, the distance between the user's gaze point and the center of each sub-view in the entire light field array may be minimized. The sub-view associated with that minimum distance may be selected as the viewpoint sub-view.

[0216] In some embodiments, the selection of a representation (e.g., a subsampling view of a light field array) may be based on the user's position. In some embodiments, the viewing client may track the user's head position, and the selection of a representation may be based on the tracked user head position. In some embodiments, the direction of the user's gaze may be tracked, and the selection of a representation may be based on the tracked direction of the user's gaze.

[0217] FIG. 11 is a message sequencing diagram illustrating an exemplary process for multiplexed light field rendering using an exemplary client pool model according to some embodiments. In some embodiments of the exemplary light field (LF) generation process (1100), as described for some examples in relation to FIG. 2 through 9, the server (1102) may generate a subsampled light field and description for the MPD (1106). In some embodiments, the client (1104) (or viewing client) may track the viewer's gaze (1108) and may use the viewer's gaze to select content. A content request may be sent from the client (1104) to the server (1102) (or content server) (1110), and an MPD may be sent in response to the client (1104) (1112). In some embodiments, the server (1102) may send the MPD to the client without a request (1112). In some embodiments, the client (1104) may estimate the available bandwidth for the network between the server (1102) and the client (1104) (1114) and / or estimate the bandwidth to be used under one or more subsampling scenarios. The client (1104) may select light field subsampling based in part on the estimated bandwidth(s) (1116). For example, LF subsampling may be selected from a set of samples that minimizes the bandwidth used while maintaining one or more light field parameters that exceed a threshold (e.g., using a resolution and bit rate that exceed a specific threshold for the top two priority subviews) (1116). The client (1104) may send a light field representation request to the server (1102) (1118). The server (1102) may retrieve the requested LF representation and send the requested representation to the client (1120).The client (1104) may receive the requested representation as one or more content segments. In some embodiments, the client (1104) may interpolate one or more interpolated subviews using the received content segments (1124). The viewing client (1104) may display the received and interpolated subview light field content data (1126). In some embodiments, one or more composite views (e.g., interpolated composite views) may be composited from one or more interpolated subviews, and the viewing client (1104) may display one or more composite views (e.g., interpolated composite views) (1126).

[0218] For example, the client may receive content segments for the highest, second highest, and third highest priority subviews illustrated in FIG. 4, and the client may interpolate content data for one or more additional subviews. In some embodiments, for example, content segments may be requested by the client for the highest priority subview at the request rate indicated in Equation 1:

[0219]

[0220] For variable x, which is the same as the unit of time, the content segment may also be requested by the client for the subview with the second highest priority at the request rate indicated in Equation 2:

[0221]

[0222] The content segment may also be requested by the client for the subview with the third highest priority at the request rate indicated in Equation 3:

[0223]

[0224] Some embodiments may interpolate subview content data for subviews for which content data has not been received during a specific time step. Some embodiments may store subview content data and may use stored subview content data for time steps occurring within subview request rates for a specific priority. The request rates indicated in Equations 1 through 3 are examples, and the request rates may be assigned differently in some embodiments.

[0225] In some embodiments, the selection of a light field representation (which may be a light field subsampling in some embodiments) may be based on bandwidth constraints. For example, a representation that falls under bandwidth constraints may be selected. In some embodiments, the user head position may be tracked, and a representation may be selected based on the user head position. In some embodiments, the direction of the user's gaze may be tracked, and a representation may be selected based on the direction of the user's gaze. In some embodiments, the client process may include interpolating at least one view, and as a result, selecting a representation selects a representation from a group containing the interpolated view. In some embodiments, the client process may include requesting light field video content from a server, and as a result, obtaining a selected subsampled representation includes executing a process selected from the group consisting of: retrieving a selected subsampled representation from a server, requesting a selected subsampled representation from a server, and receiving a selected subsampled representation. In some embodiments, the client process may include requesting a media manifest file from the server; and requesting light field content associated with a selected view subset, and as a result, obtaining light field content associated with the selected view subset may include executing a process selected from the group consisting of: retrieving light field content associated with the selected view subset from the server; requesting light field content associated with the selected view subset from the server; and receiving light field content associated with the selected view subset.

[0226] FIG. 12 is a message sequencing diagram illustrating an exemplary process for multiplexed light field rendering using an exemplary server push model according to some embodiments. Client and server processing is described in more detail in the application.

[0227] In some embodiments, the exemplary server push process (1200) may include, for example, a preprocessing process and a runtime process (1222). In some embodiments of the exemplary preprocessing process, the viewing client (1202) may send a content request to the content server (1204) (1206), and the content server (1204) may respond using the first full light field frame (1208). The viewing client (1202) may update the viewpoint (1210) and composite the view, which may be done using the first full light field frame.

[0228] In some embodiments of the exemplary runtime process (1222), the viewing client (1202) may transmit an updated viewpoint to the content server (1212). The content server (1204) may predict the viewpoint position at the next time step using a motion model of the viewpoint movement. In some embodiments, the content server (1204) may construct a motion model using the partially tracked movement of the user (1214). The content server (1204) may analyze the content (1216) and prioritize the light field sub-views to be rendered for the next time step based on the predicted view position. The content server (1204) may transmit the selected light field sub-view, the virtual camera position associated with the selected sub-view, and the timestamp of the content to the viewing client (1202) (1218). The viewing client (1202) may update the viewpoint (1220) and synthesize one or more sub-views using the temporal variability of the light field data. In some embodiments, the frame may include a frame-packed representation of two or more views corresponding to a selected representation. In some embodiments, the light field data may include 2D image frame data transmitted from a content server to a client device. In some embodiments, the light field content may include a frame of the light field content (which may also be transmitted from a content server to a client device).

[0229] FIG. 13 is a message sequencing diagram illustrating an exemplary process for multiplexed light field rendering using an exemplary client pool model according to some embodiments. In some embodiments of the exemplary client pool model process (1300), the client (1302) may perform future viewpoint predictions and may perform content analysis to determine which sub-views are prioritized. In some embodiments, based on the priority assigned to sub-views according to the analysis performed by the viewing client (1302), the client (1302) may retrieve a specific sub-view stream from the content server (1304). In this model, the client (1302) may select which light field sampling to use by selecting how many sub-views to retrieve and by selecting which spatial distribution to use across the entire light field array. The viewing client (1302) may dynamically change the number of individual sub-views pulled and the sampling distribution according to viewpoint movement, content complexity, and display capability. A subset of views may be collected from a server or, in some embodiments, provided as an alternative light field subsampling. The client may select one of these subsets based on local criteria such as available memory, display rendering capability, and / or processing power. Distributing a subset of views may allow the encoder to utilize redundancy between views for greater compression compared to compressing views individually. FIG. 13 illustrates an overview of the sequence for processing and communication between a viewing client (1302) and a content server (1304) in an exemplary client pull session (1300).

[0230] In some embodiments, the content server (1304) may render a subset of light fields in sequential order (1306). The viewing client (1302) may request content (1308), and the content server (1304) may resend the MPD file (1310). The viewing client (1302) may select a subset to be retrieved (1312). The viewing client (1302) may request a subset (1314), and the content server (1304) may respond with the subset content data, a virtual camera position, and a timestamp capturing the content data (1316). The viewing client (1302) may store the received data in a local cache (1318). The viewing client (1302) may update the user's viewpoint (1320) and composite the view using the temporal variability of the light field data stored in the local cache. The viewing client (1302) may use a motion model of the viewpoint movement (or construct one in some embodiments (1322)) and predict the viewpoint position at the next time step. The viewing client (1302) may analyze the content (1324) and prioritize a subset of light fields to be brought in based on the predicted viewpoint position.

[0231] In some embodiments of the exemplary client pool model, the client receives a media presentation description from the content server. The media presentation description (MPD) may represent an available subset or may include alternative light-field subsampling compiled by the content server. The subset may include individual subviews or a collection of subviews, and the sampling density, distribution, and the number of subviews included may vary for each subset. By compiling multiple subviews into a single subset, the content server may provide good candidate subsampling depending on the content. The encoder may also leverage redundancy between views for greater compression when the subset is delivered as a single stream. Each subset may be available in one or more spatial resolutions and compression versions.

[0232] In some embodiments of the client pool model, the content server may render and / or provide each individual light field subview at a fixed frame rate. In some embodiments, the content server may analyze the content to determine the frequency of subview updates. In some embodiments, selecting a presentation may include predicting the user's viewpoint; and selecting a selected presentation may be based on the user's predicted viewpoint.

[0233] FIG. 14 is a flowchart illustrating an exemplary process for a content server according to some embodiments. In some embodiments, an exemplary process (1400) executed by the content server may include receiving a content request from a client (1402) (e.g., “waiting for a content request from a client”). A viewing client may use tracking (e.g., user and / or eye position tracking) to determine the user viewpoint and to transmit the user viewpoint to the content server when displaying the content. The process (1400) may further include rendering an entire aggregate image for a first sequence of frames of the aggregate light field (1404). Spatial data (1406) may be retrieved from memory, such as a cache local to the server. The rendered light field data may be transmitted to the client (1408). The client's current viewpoint may be received (1410) and stored in memory as viewpoint data (1412). A motion estimation model may be constructed and / or updated using viewpoint location data received from a client (1414). The predicted viewpoint location for the next time step (which may be the same as the estimated time of arrival of the rendered content at the viewing client) may be estimated using a motion estimation model. In some embodiments, the predicted viewpoint location may be based on a model that estimates the network communication latency between the server and the client and the continuity of the viewpoint motion. To estimate the motion, Kalman filtering or any other suitable motion estimation model may be used. The content may be analyzed using the predicted viewpoint for the next time step to determine the rendering priority for the light field subview (1416).Using the predicted time point, the content server may perform content analysis to determine the highest priority subview at the next time step to render and transmit to the viewing client. Based on the rendering priority, the subview may be rendered (1418). The rendered light field subview content, timestamp, and virtual camera position may be streamed to the client (1420). A determination may be made as to whether the session is terminated (1422). If the session is not terminated, the process (1400) may be repeated, starting with receiving the client's current time point (1410). A determination may be made as to whether the processing is terminated (1424). If the termination of the processing signal is received, the process (1400) is terminated (1426). Otherwise, the process (1400) may be repeated, starting with waiting for a content request from the client (1402). In some embodiments, for the duration of the content or until the viewing client signals the end of the session, the content server continuously streams multiplexed light field data that prioritizes the rendering of subviews.

[0234] FIG. 15 is a flowchart illustrating an exemplary process for a viewing client according to some embodiments. When a user launches a viewing client application, the user may also indicate content to be viewed. The content may be a link to the content, for example, located on a content server and / or streamed from a content server. The link to the content may be a URL identifying the content server and the specific content. The viewing client application may be launched automatically by an operating system or by an explicit command by the user based on a content type request and identifying the application associated with the specific content type. In addition to being a standalone application, the viewing client may be integrated with a web browser or social media client, or the viewing client may be part of an operating system. When the viewing client application is launched, the viewing client may initialize the device, the user, and / or the sensor used for eye tracking.

[0235] Some embodiments of an exemplary process (1500) executed by a viewing client may include requesting content from a content server (1502) and initializing user eye tracking (1504). The viewing client may receive a first full integrated light field image frame from the content server (1506). The light field data may be stored in a local cache memory (1508) or displayed by the viewing client. The initial user viewpoint may be set using a default position that may be used to display the first light field frame. The viewpoint may be updated based on device tracking and user input (1510). The current viewpoint may be transmitted to the content server (1512). The content server may use the received viewpoint to render one or more subviews of the light field that may be streamed to the client. The client may also receive these subviews along with a timestamp indicating a shared time step of rendering, and the position of the virtual camera for the entire array of images and optical parameters used to render the subviews (1518).

[0236] One or more views of a light field may be composited based on the current view (1514). The composited view may be a new view that does not contain light field content specifically associated with that particular view, for example. The composited view(s) may be displayed by a viewing client (1516). The viewing client may receive light field sub-view content rendered by a content server, which may be stored in memory such as a local light field cache (1508) (1518). The viewing client may composite a view for one or more rendering steps using the latest view (1510) which is updated using traces and user input. A decision may be made to determine whether a termination of processing signal is received (1520). If a termination of processing signal is received, the process (1500) may be terminated (1522). Otherwise, the process (1500) may be repeated by updating the user view (1510).

[0237] In some embodiments, the client process may include generating a signal for display using the rendered representation. In some embodiments, the client process may include determining the user's viewpoint using the direction of the user's gaze, and consequently, selecting a selected representation may include selecting a selected representation based on the user's viewpoint. In some embodiments, the client process may include determining the user's viewpoint using the direction of the user's gaze; and selecting at least one subsampling of a view of a multiview video, and consequently, selecting one subsampling of a view may include selecting one subsampling of a view within a threshold viewpoint angle of the user's viewpoint.

[0238] In some embodiments, the viewing client renders the image to be displayed by synthesizing a viewpoint that matches the current viewpoint using locally cached light field data. When synthesizing a viewpoint, the viewing client may also select a subview to be used in the synthesis. In some embodiments, for selection, the viewing client may examine the local light field cache to identify a subview that, for example, has the most recent available data close to the current viewpoint, and has sufficient data from various time steps to enable interpolation or prediction to generate a good estimate of the subview appearance at a given time step, etc. To mitigate temporal variation between subviews (which may be received from separate subset streams), the viewing client may use a technique developed for video frame interpolation, such as Nikilaus, S. et al.'s *Video Frame Interpolation via Adaptive Separable Convolution, PROC. OF THE IEEE INT'L CONF. ON COMP. Using the technology mentioned in 『VISION 261-270 (2017)』("Nikilaus"), if there are "future" frames already available from the content server, a frame corresponding to the rendering time may be generated, or if only frames from an earlier time step available for a specific subview exist Video frame predictions may also be generated as described in *One-Step Time-Dependent Future Video Frame Prediction with a Convolutional Encoder-Decoder Neural Network, PROC. OF INT'L CONF. ON IMAGE ANALYSIS AND PROCESSING 140-151 (2017)* by Vedran et al. In some embodiments, the viewing client may use one of these two methods to estimate a subview for a specific time step that may be used to render and display the view. In some embodiments, the viewing client uses a subview image stored in a local cache for several time steps. In some embodiments, where the viewing client has all subviews to be used to synthesize a viewpoint at a given time step, the viewing client may synthesize a new viewpoint from, for example, a coarse, aggregated light field formed by a selected subview by using the process described in “Learning-Based View Synthesis for Light Field Cameras, 37.6 ACM TRANSACTIONS ON GRAPHICS (TOG) 193 (2016)” by Kalantari, Nima Khademi et al. (“Kalantari”).

[0239] In some embodiments, an exemplary viewing client process may include determining the user's viewpoint from the direction of the user's gaze. The process may further include selecting one or more subviews of light field video content from a group comprising one or more interpolated views and retrieved subsampled representations. The process may further include compositing a view of the light field using the selected one or more subviews and the user's viewpoint. In some embodiments, the process may include determining the user's viewpoint from the direction of the user's gaze; and selecting one of a plurality of subsampled representations based on the user's viewpoint. In some embodiments, an exemplary viewing client process may further include displaying the composite view of the light field. In some embodiments, selecting one or more subviews of the light field may include selecting one or more subviews within a threshold viewpoint angle of the user's viewpoint.

[0240] In some embodiments, the client process may include determining the user's viewpoint from the direction of the user's gaze, and as a result, selecting one of a plurality of subsampled representations may additionally be based on the user's viewpoint. In some embodiments, the client process may include displaying a composite view of the light field. In some embodiments, selecting one or more subviews of the light field may include selecting one or more subviews within a threshold viewpoint angle of the user's viewpoint.

[0241] Adaptive Light Field Streaming

[0242] Supporting dynamic user viewpoints and motion parallax using many existing multiview coding standards leads to excessive bandwidth usage, particularly in applications that generate only a single view or stereo view 2D image for display at a single time step. Examples of such applications include 3 DoF+ or 6 DoF applications based on viewing content on an HMD or other 2D display.

[0243] In some embodiments, to optimize rendering and data distribution, a subset of the entire aggregated light field data may be generated and transmitted, and the receiver may synthesize additional views from the transmitted views. Furthermore, the influence of perceived image quality resulting from the selection of rendered and transmitted subviews may guide the process of subview rendering on either the server side or the client side to ensure experience quality.

[0244] In some embodiments, data transmission may be optimized by adapting light field streaming to available resources by changing the number of light field subviews during streaming.

[0245] The content server may provide multiple versions of light field data featuring a variable number of light field subviews. In addition to multiple versions of the streamed content, the server may provide metadata describing the available streams as an adaptive streaming manifest. At the start of a content streaming session, the client may download the manifest metadata and initiate the download of content segments based on the manifest metadata. While downloading content segments, the client may observe session characteristics and network and processing performance, and may adapt the streaming quality by switching between available content streams.

[0246] In some embodiments, the client may optimize the quality of experience by dynamically adapting light field streaming to various performance and network transmission conditions. Content delivery bandwidth requirements may be reduced by allowing the viewing client to dynamically adjust the number of light field subviews transmitted from the content server to the viewing client.

[0247] Some embodiments may use an exemplary client pool model that operates similarly to the MPEG-Dash model for streaming. The server may generate multiple versions of light field content by changing the number of subviews included in each single version of the stream. The server may provide the client with a manifest file (e.g., MPD) indicating the number and location of subviews included in each available version of the content stream. The client may continuously analyze session characteristics and performance metrics to determine which stream to download in order to provide Quality of Experience (QoE) using given network and computing resources.

[0248] Supporting dynamic viewpoints and motion parallax using existing multiview coding standards leads to excessive bandwidth usage, particularly in applications that require only a smaller number of viewpoints at any given time. Exemplary methods and systems according to some embodiments may avoid existing limitations by considering the impact on the perceived image quality of a selected number of subviews packaged into frames for current and predicted user viewpoints and delivery.

[0249] FIG. 16 is a schematic example illustrating an exemplary light field rendered with 5×5 sub-views according to some embodiments. In some embodiments, subsampling of the views may be used to reduce transmission bandwidth in conjunction with view interpolation used at the receiver to generate the required views. In some embodiments, changing the number of packed views may be used to adapt to bandwidth or processing constraints at the client. FIG. 16 illustrates an exemplary entire light field (1600) rendered using a 5×5 virtual camera array of sub-views (1602).

[0250] FIG. 17 is a schematic example illustrating an exemplary light field rendered with 3×3 subviews according to some embodiments. The priority for individual light field subviews may be determined based on content analysis and context information (e.g., user / gaze position and display capability). A set of individual subviews may be generated based on sequential priority. A client may receive subviews, store subviews in a cache, and synthesize new viewpoints using temporally coherent interpolation of subviews. FIG. 17 shows an example of nine views packed in a 3×3 grid (1700). The views of FIG. 17 were selected in the form of a checkerboard pattern from the entire light field illustrated in the 5×5 array of FIG. 16 using the priority as described above for some embodiments. FIG. 19b presents (below) an example in which specific subviews are selected for an exemplary 3×3 array.

[0251] FIG. 18 is a schematic example illustrating an exemplary light field rendered with 2×3 subviews according to some embodiments. FIG. 18 shows an example of six views packed in a 2×3 grid (1800). The views of FIG. 18 were selected in a rectangular pattern from the entire light field illustrated in the 5×5 array of FIG. 16 using the priority as described above for some embodiments. FIG. 19c presents (below) an example in which specific subviews are selected for an exemplary 2×3 array.

[0252] FIG. 19a is a diagram illustrating an exemplary 5×5 array light field configuration corresponding to FIG. 16 according to some embodiments. In some embodiments, the Media Presentation Description (MPD) may include details of the entire (or complete) light field array, such as the number of views in the adaptation set, indices for those views, and sampling positions of the views. For example, the entire light field array (1900) may have N×M view positions. FIG. 19a illustrates an exemplary naming convention for the entire light field 5×5 array (1900). The first number at each coordinate position indicates the row, and the second number indicates the column. Position (1, 1) is at the top-left corner, and position (5, 5) is at the bottom-right corner. Other embodiments may use a different naming convention, such as the example illustrated in FIG. 21.

[0253] FIG. 19b is a diagram illustrating an exemplary 3×3 array light field configuration corresponding to FIG. 17 according to some embodiments. The 3×3 array (1920) shown in FIG. 19b corresponds to the solid black ellipse in FIG. 20. For example, the uniformly subsampled 3×3 array (1920) in FIG. 19b represents a sub-view (5, 3) in the bottom center position. This sub-view corresponds to the black ellipse in FIG. 20 in the 5th (bottom) row and the 3rd (center) column.

[0254] FIG. 19c is a diagram illustrating an exemplary 2×3 array light field configuration corresponding to FIG. 18 according to some embodiment. The 2×3 array (1940) illustrated in FIG. 19c corresponds to a rectangular pattern of views selected from the middle portion of the entire light field of the 5×5 array of FIG. 16. For example, the 2×3 array (1940) of FIG. 19c represents sub-views (2, 4) in the top-right position. These sub-views correspond to the second (middle-top) row and the fourth (center-right) column of FIG. 16. The exemplary 2×3 array (1940) of FIG. 19c may also be illustrated using black ellipses, as in FIG. 20 (not illustrated).

[0255] FIG. 20 is a schematic example illustrating exemplary subsampling using 3×3 subviews according to some embodiments. A content server may generate several subsampled versions of the original light field data by using several subsampling configurations. For example, the content server may reduce the number of subviews of the entire light field (2000) to subsampling of 3×3 subviews (2002, 2004, 2206, 2208, 2010, 2012, 2014, 2016, 2018), as exemplified in FIG. 17 and FIG. 19b. Similarly, the content server may generate, for example, 10×10, 5×5, 2×2, and 1×1 array subsamplings.

[0256] The content server may also generate metadata describing the streams available in the MPD file. For example, if the MPD describes two subsampling configurations (a 10×10 array and a 5×5 array), the 10×10 array may use the first stream and the 5×5 array may use the second stream. In some embodiments, support for the 10×10 array of subviews may, for example, have only one stream (not 100 individual streams). Similarly, in some embodiments, the 5×5 array may have only five streams (or a different number of streams, fewer than 25).

[0257] FIG. 21 is a schematic example illustrating an exemplary numbering configuration of light field subviews according to some embodiments. The exemplary subview numbering configuration (2100) illustrated in FIG. 21 is used in the exemplary MPD illustrated in Table 2. FIG. 21 illustrates an example of a subview numbering configuration for an entire light field 5×5 array. The subview numbering configuration of FIG. 21 may be compared with the subview numbering configuration of FIG. 19a. While FIG. 19a illustrates an exemplary coordinate subview numbering configuration, FIG. 21 illustrates an exemplary sequential subview numbering configuration. The top row of the 5×5 array in FIG. 21 numbers subviews 1 through 5 from left to right. The second row numbers subviews 6 through 10 from left to right and continues to the last row, numbering subviews 21 through 25 from left to right. FIGS. 21 through 23 and Table 2 together illustrate exemplary MPDs for an exemplary client pool model.

[0258] FIG. 22 is a diagram illustrating an exemplary MPEG-DASH Media Presentation Description (MPD) file according to some embodiments. An exemplary client pool model may use the standard structure of the MPEG-DASH Media Presentation Description (MPD) illustrated in FIG. 22. The MPD file format illustrated in FIG. 22 may be used to transmit the entire media description downloaded by a viewing client as part of a streaming session initialization. The organizational structure of the MPEG-DASH MPD is illustrated in FIG. 22.

[0259] The top-level duration fields (2204, 2206) may represent a start time and a duration. The MPD (2202) may include one or more duration fields (2204, 2206). The duration fields (2204, 2206) may include one or more adaptation sets (2208, 2210). The adaptation sets (2208, 2210) may include one or more representation fields (2212, 2214). Each representation (2212, 2214) within the adaptation sets (2208, 2210) may contain the same content encoded using different parameters. The representation fields (2212, 2214) may include one or more segments (2216, 2218). The segments (2216, 2218) may include one or more sub-segments (2224, 2226) containing DASH media files. The expression field (2212, 2214) may be divided into one or more sub-expression fields (2220, 2222). The sub-expression field (2220, 2222) may contain information applicable to only one media stream.

[0260] FIG. 23 is a diagram illustrating an exemplary MPD file having light field configuration elements according to some embodiments. FIG. 23 illustrates a method in which MPD data may be organized according to the MPEG-DASH protocol structure to enable an exemplary client pool model of light field streaming.

[0261] In some embodiments, the MPD structure (2300) uses periods (2304, 2306) as top-level entities. Each period (2304, 2306) may provide information about a single light field scene. A single scene may be, for example, a continuous light field rendering in which a virtual camera array used for rendering is kept constant. The entire experience may include multiple scenes, each specified in a separate period block. Each period block may include a light field rendering setup that is labeled as a light field description (2308) in FIG. 23 and corresponds to a first adaptation set. The light field rendering setup block (2308) may include the number of sub-views, the placement of virtual cameras used by the sub-views, and an overview of the scene layout, such as the size of the views and the size and placement of elements in the views. Each period block (2304, 2306) may include one or more subset blocks (2310, 2312, 2314), each subset block corresponding to an adaptation set. Each subset adaptation set (2310, 2312, 2314) may include one or more different versions of content encoded using different configuration parameters. For example, the content may be encoded using different resolutions, different compression rates, different bitrates, and / or different supported codecs. Each subset (2310, 2312, 2314) may include a selected number of subviews within different adaptation sets having resolutions according to the representation block, encoded using a specific codec at a specific bitrate for the segment block. Each subset (2310, 2312, 2314) may be divided into short clips contained in subsegment blocks having links to actual video data.

[0262] An adaptation set within the period (2304, 2306) of the MPD (2302) may include a subset (2310, 2312, 2314) of the entire array of views that changes the number of views and the sampling positions of the views within the adaptation set. The adaptation set may include the number of views that exist and the index of the available views. The adaptation set may indicate the priority level of the subviews. An adaptation set (or, in some embodiments, a subset (2310, 2312, 2314)) may include one or more resolutions (2318, 2320) each containing one or more bitrates (2322, 2324, 2326). For each resolution (2318, 2320), there may be a series of time steps 1, 2, ..., N (2328, 2330, 2332). Each time step (2328, 2330, 2332) may have a separate URL (2334, 2336, 2338, 2340, 2342, 2344) for each supported bitrate. In the example illustrated in FIG. 23, there are N resolutions each supporting N bitrates. Additionally, an adaptation set within the periods (2304, 2306) of the MPD (2302) may be used for audio, as illustrated as an audio block (2316) in the example of FIG. 23.

[0263] In some embodiments, the MPD may include information corresponding to two or more views corresponding to a selected view subset. In some embodiments, for at least one of the view subsets, the media manifest file (e.g., MPD) may include information corresponding to two or more views of light field content.

[0264] Table 2 shows pseudocode for an exemplary MPD having sub-view configurations for a field as shown in FIG. 23 and a 5×5 array light field as shown in FIG. 21. The example shown in Table 2 illustrates three different sets of light field view adaptations having varying numbers of views packed into each frame. Traditional DASH rate and resolution adaptations may also be used within each view category.

[0265] period

[0266] o Adaptation Set 1: Light Field Scene Description:

[0267] Spacious view array. Number of 5×5 subviews. Virtual camera position.

[0268] o Adaptation Set 2: Subset 1 (Whole Light Field): Numbers of Sub-views 5×5 (Sub-views include 1-25)

[0269] Resolution 1: 1920×1080 pixels per subview

[0270] Bitrate 1: Encoded using Codec XX, Required transmission capacity 35 Mbps

[0271] Bitrate 2: Encoded using Codec XX, Required transmission capacity 31 Mbps

[0272] Resolution 2: 1280×720 pixels per subview

[0273] Bitrate 1: Encoded using Codec XX, Required transmission capacity 22 Mbps

[0274] Bitrate 2: Encoded using Codec XX, Required transmission capacity 19 Mbps

[0275] o Adaptation Set 3: Subset 2: Numbers of the 3×3 sub-views (sub-views include 1, 3, 5, 11, 13, 15, 21, 23, 25)

[0276] Resolution 1: 1920×1080 pixels per subview

[0277] Bitrate 1: Encoded using Codec XX, Required transmission capacity 10 Mbps

[0278] Bitrate 2: Encoded using Codec XX, Required transmission capacity 9 Mbps

[0279] Resolution 2: 1280×720 pixels per subview

[0280] Bitrate 1: Encoded using Codec XX, Required transmission capacity 7 Mbps

[0281] Bitrate 2: Encoded using Codec XX, Required transmission capacity 6 Mbps

[0282] o Adaptation Set 4: Subset 3: Numbers of the 2×2 sub-views (sub-views include 1, 5, 21, and 25)

[0283] Resolution 1: 1920×1080 pixels per subview

[0284] Bitrate 1: Encoded using Codec XX, Required transmission capacity 7 Mbps

[0285] Bitrate 2: Encoded using Codec XX, Required transmission capacity 6 Mbps

[0286] Resolution 2: 1280×720 pixels per subview

[0287] Bitrate 1: Encoded using Codec XX, Required transmission capacity 4 Mbps

[0288] Bitrate 2: Encoded using Codec XX, Required transmission capacity 3 Mbps

[0289] o Adaptation Set 5: Subset 4: Number of Sub-view 1×1 (Sub-views include 13)

[0290] Resolution 1: 1920×1080 pixels per subview

[0291] Bitrate 1: Encoded using Codec XX, Required transmission capacity 2 Mbps

[0292] Bitrate 2: Encoded using Codec XX, Required transmission capacity 1.5 Mbps

[0293] Resolution 2: 1280×720 pixels per subview

[0294] Bitrate 1: Encoded using Codec XX, Required transmission capacity 1 Mbps

[0295] Bitrate 2: Encoded using Codec XX, Required transmission capacity 0.7 Mbps

[0296] Table 2.

[0297] FIG. 24 is a system diagram illustrating an exemplary set of interfaces for adaptive light field streaming according to some embodiments. In some embodiments, an exemplary system (2400) having a server (e.g., a content server (2402)) may include a database (2410) of light field content and one or more MPDs. The content server (2402) may execute an exemplary process (2412) for preprocessing the content. One or more content servers (2402) may be connected to a client (e.g., a viewing client (2404)) via a network (2418). In some embodiments, the viewing client (2404) may include a local cache (2416). The viewing client (2404) may be enabled to execute an exemplary process (2414) for compositing a view (or a sub-view in some embodiments) into the content. A viewing client (2404) may be connected to a display (2406) to be used for displaying content (including a composite view of the content). In some embodiments, the viewing client (e.g., 2404) may include an input (2408) for receiving tracking data (e.g., user eye tracking data) and input data. For example, the exemplary content server (2402) of FIG. 24 may be configured similarly to, for example, the exemplary content server of FIG. 10, or may be configured differently.

[0298] FIG. 25 is a message sequencing diagram illustrating an exemplary process for adaptive light field streaming using estimated bandwidth and view interpolation according to some embodiments. In some embodiments of the exemplary adaptive LF streaming process (2500), as described for some examples in relation to FIG. 16 through 23, the server (2502) may generate a description of the MPD and a subsampled light field (2506). In some embodiments, the client (2504) (or viewing client) may, for example, track the viewer's gaze (2508) and use the viewer's gaze to select content. In some embodiments, for example, tracking the user's position (e.g., in combination with the gaze, or instead of the gaze) may be used. A content request may be transmitted from the client (2504) to the server (or content server (2502)) (2510), and an MPD may be transmitted in response to the client. In some embodiments, the server (2502) may transmit the MPD to the client without a request (2512). In some embodiments, the client (2504) may estimate the available bandwidth for the network between the server (2502) and the client (2504) (2514) and / or estimate the bandwidth to be used under one or more subsampling scenarios. The client (2504) may select a quantity of packed views or an array configuration based in part on the estimated bandwidth(s) (2516). For example, the quantity of packed views or an array configuration may be selected from a sample set that minimizes the bandwidth used while maintaining one or more light field parameters that exceed a threshold (e.g., using a resolution and bit rate that exceed a specific threshold for the top two priority sub-views) (2516). The client (2504) may transmit a light field representation request to the server (2502) (2518).The server (2502) may retrieve the requested LF expression (2520) and transmit the requested expression to the client (2504) (2522). The client (2504) may receive the requested expression as one or more content segments. In some embodiments, the client (2504) may interpolate one or more interpolated subviews using the received content segments (2524). The viewing client (2504) may display the received and interpolated subview light field content data (2526).

[0299] In some embodiments, the client process may include: obtaining light field content associated with a selected representation; decoding a frame of light field content; combining one or more views represented in the frame to generate a combined view composite result; and rendering the combined view composite result to a display.

[0300] FIG. 26 is a message sequencing diagram illustrating an exemplary process for adaptive light field streaming using predicted view positions according to some embodiments. In some embodiments, the client (2602) may perform future viewpoint predictions and may perform content analysis to determine which sub-views are prioritized. In some embodiments, prioritization may be based, for example, on delivery bandwidth and view interpolation requirements and resources. In some embodiments, for example, based on the priority assigned to sub-views according to analysis performed by the viewing client (2602), the client (2602) may retrieve a specific representation featuring a selected number of sub-views from the content server. In this model, the client (2602) may select which light field sampling to use by selecting how many sub-views to retrieve. The viewing client (2602) may dynamically change the number of individual sub-views retrieved according to viewpoint movement, content complexity, and display capability. Content representations having different numbers of views may be collected from the server or provided as alternative light field subsampling for some embodiments. The client (2602) may select one of these subsets based on local criteria such as available memory, display rendering capability, and / or processing power. FIG. 26 illustrates an overview of the sequence for processing and communication between the viewing client (2602) and the content server (2604) in an exemplary client pool session (2600).

[0301] In some embodiments, the content server (2604) may render a subset of light fields in sequential order (2606). The viewing client (2602) may request content (2608), and the content server (2604) may resend the MPD file (2610). The viewing client (2602) may select the number of views of the content representation to be retrieved (2612). The viewing client (2602) may request a subset (2614), and the content server (2604) may respond with the subset content data, a virtual camera position, and a timestamp capturing the content data (2616). The viewing client (2602) may store the received data in a local cache (2618). The viewing client (2602) may update the user's viewpoint (2620) and composite the view using the temporal variability of the light field data stored in the local cache. The viewing client (2602) may use a motion model of the viewpoint movement (or construct one in some embodiments (2622)) and predict the viewpoint position at the next time step. The viewing client (2602) may analyze the content (2624) and prioritize a light field subset to be brought in based on the predicted viewpoint position.

[0302] In some embodiments, selecting one of a plurality of view subsets may include parsing information from a media manifest file (e.g., MPD) for a plurality of view subsets of light field content.

[0303] FIG. 27 is a flowchart illustrating an exemplary process for content server preprocessing according to some embodiments. In some embodiments of the preprocessing step, the server may generate a plurality of light field data subsets and metadata describing the subsets. The plurality of light field data subsets may be configured by selecting the quantity of subviews in various subsamplings to be used to reduce the number of subviews included in the streaming data.

[0304] In some embodiments, the content server may execute an exemplary preprocessing method multiple times for different subsampling configurations. In some configurations of the exemplary preprocessing method, the original light field data (2704) may be received from or read from memory, such as a local content server cache. The content server may select a variation having a varying number of subviews (2708) to be generated (2702). The content server may generate streaming data for each subview configuration (or subsampling configuration for some embodiments) (2706). The content server may generate an MPD (or metadata within the MPD) for each version or variation of the streaming data (2710). The MPD (2712) may be stored in a local cache by the content server.

[0305] FIG. 28 is a flowchart illustrating an exemplary process for content server runtime processing according to some embodiments. In some embodiments, an exemplary adaptive content distribution runtime process (2800), such as the example illustrated in FIG. 28, may be executed after performing preprocessing, such as the exemplary preprocessing method of FIG. 27. In some embodiments, the content server may receive a request for content from a viewing client (2802). The runtime process (2800) may determine the request type (2804). If the content request represents a new session, the content server may retrieve an MPD corresponding to the content request from memory (e.g., local cache) (2806) and transmit the MPD to the viewing client (2808). In some embodiments, the MPD may be generated at runtime, which may be similar to the process described in connection with FIG. 27, for example. If the content request indicates a request for a subset data segment, the content server may retrieve a light field subset stream from memory (2812) and send the data segment to the viewing client (2810). The runtime process (2800) may determine whether the termination of processing is requested (2814). If the termination of processing signal is received, the process may be terminated (2816). Otherwise, the process may be repeated by waiting to receive a request for content (2802).

[0306] FIG. 29 is a flowchart illustrating an exemplary process for a viewing client according to some embodiments. In some embodiments of the exemplary viewing client process (2900), a content request may be sent to a content server (2902). In some embodiments, a content request may be sent when a user launches a viewing client application and indicates content to view. In some embodiments, the viewing client application may be automatically launched by an operating system by executing an explicit command by the user or a script file, or by identifying an application associated with a content request and the type of content within the content request. In some embodiments, the viewing client may be, for example, a standalone application, an integrated web browser or social media client, or part of an operating system. In some embodiments, launching the viewing client application causes initialization of a sensor, such as a sensor used for eye tracking, for example, by the viewing client device, by the user.

[0307] The viewing client may receive a manifest file (e.g., a media manifest file), such as an MPD (or an adaptation manifest in some embodiments) corresponding to the requested content, for example (2904). In some embodiments, the viewing client initializes device tracking (2906). The viewing client may select an initial number of subviews (or representations) to request from the content server (or to download from the content server) (2908). The viewing client may download subsegments sequentially from the content server. In some embodiments, the viewing client may determine an initial adaptation set (or a number of packed subsets in some embodiments) and a specific representation to request from the content server, for example, based on display settings, tracking application settings, resolution, and bitrate.

[0308] Subviews may be packed into each frame of the content to be retrieved by the viewing client. In some embodiments, the viewing client may request a stream having a selected number of subviews (2910). In some embodiments, the viewing client may download media segments for a selected subset from a content server. In some embodiments, the content may be a URL link identifying content data and / or MPD files. The viewing client receives a set of subviews (2912). Upon receiving a first segment of the presentation, the client may, in some embodiments, initiate a successive runtime process. The runtime process may include updating the viewpoint (2914). The viewpoint may be updated based on device tracking and user input. The viewpoint may be synthesized using the received light field data and the current viewpoint (2916). From the received data, the client renders the light field in a display-specific format and interpolates additional subviews for some viewing conditions.

[0309] The number of selected subviews may be updated based on, for example, user tracking (e.g., user position, user gaze, user viewpoint, user view tracking), content analysis, performance metrics, viewpoint movement, content complexity, display capability, network and processing capability, or any other suitable criteria (2918). In some embodiments, performance metrics (e.g., bandwidth used, available bandwidth, and client capability) may be measured or observed and the number of selected subviews may be updated. A viewing client may update the number of selected subviews as part of an exemplary runtime process. The runtime process (2900) may include determining (2920) whether termination of processing is requested. If termination of processing signal is received, the process (2900) may be terminated (2922). Otherwise, the process (2900) may be repeated by requesting (2910) the selected subview stream (or downloading in some embodiments).

[0310] In some embodiments, the frame may include a frame-packed representation of two or more views corresponding to a selected view subset. In some embodiments, combining two or more views represented in the frame may involve using view composition technology. In some embodiments, selecting one of a plurality of view subsets selects the view subset based on at least one of the following criteria: eye tracking, complexity of content, display capability, and bandwidth available for retrieving light field content. In some embodiments, selecting one of a plurality of view subsets may include: predicting the user's viewpoint; and selecting the view subset based on the user's predicted viewpoint. For example, the viewpoint subset may be selected to be within a viewpoint angle threshold of the user's viewpoint.

[0311] In some embodiments, the selection of a representation may be based on at least one of the following criteria: user head position, eye tracking, complexity of the content, display capability, and bandwidth available for retrieving light field content. In some embodiments, the selection of a view subset may be based on at least one of the following criteria: user head position, eye tracking, complexity of the content, display capability, and bandwidth available for retrieving light field content.

[0312] In some embodiments, the frame may include a frame-packed representation of two or more views corresponding to a selected view subset. In some embodiments, combining two or more views represented in the frame may include using view composition techniques. In some embodiments, selecting a view subset may include: predicting the user's viewpoint; and selecting a view subset based on the user's predicted viewpoint. In some embodiments, generating a view from the acquired light field content may include interpolating the view from the light field content associated with the selected view subset using information within a manifest file corresponding to each view to generate the generated view.

[0313] FIG. 30 is a flowchart illustrating an exemplary process for a viewing client according to some embodiments. Some embodiments of the exemplary method (3000) of the viewing client may include receiving a media manifest file containing information about a plurality of subsampled representations of a view of light field video content (3002). In some embodiments, the media manifest file may be an MPD. The exemplary method (3000) may further include estimating the bandwidth available for streaming the light field video content (3004), selecting one of the plurality of subsampled representations (3006), and obtaining the selected subsampled representation (3008). Some embodiments of the exemplary method (3000) may include interpolating one or more interpolated subviews from the selected subsampled representation using information within the manifest file corresponding to one or more views (3010). Some embodiments of the exemplary method (3000) may further include synthesizing one or more composite views from one or more interpolated subviews (3012). Some embodiments of the exemplary method (3000) may further include displaying one or more composite views (3014).

[0314] In some embodiments, the exemplary method (3000) may further include requesting light field video content from a server. In some embodiments of the exemplary method (3000), retrieving a selected subsampled representation from a server may include requesting a selected subsampled representation from a server and receiving the selected subsampled representation. In some embodiments, the exemplary device may include a processor and a non-transient computer-readable medium storing instructions that, when executed by the processor, perform any of the methods described above. Some embodiments of the exemplary method may include: acquiring light field content associated with a selected representation; generating a view of the light field content from the acquired light field content; and rendering the generated view to a display. Some embodiments of the exemplary method may include: receiving a media manifest file containing information on a plurality of subsampled representations of the view of the light field video content; and estimating the bandwidth available for streaming the light field video content. Selecting one of a plurality of subsampled representations; acquiring the selected subsampled representation; interpolating one or more interpolated subviews from the selected subsampled representation using information in a manifest file corresponding to each of one or more views; synthesizing one or more synthesized views from one or more interpolated subviews; and displaying one or more synthesized views. Some embodiments of the device may include a processor; and a non-transient computer-readable medium storing instructions that, when executed by the processor, operate to perform any one of the exemplary methods listed above.

[0315] FIG. 31 is a flowchart illustrating an exemplary process for a viewing client according to some embodiments. Some embodiments of the exemplary method (3100) of the viewing client may include receiving a media manifest file containing information about a plurality of view subsets of light field content (3102). The exemplary method (3100) may further include selecting one of the plurality of view subsets (3104) and obtaining light field content associated with the selected view subset (3104). In some embodiments, the exemplary method (3100) may include decoding a frame of light field content (3108). The exemplary method (3100) may include combining two or more views represented in a frame to generate a combined view composite result (3110). The exemplary method (3100) may further include rendering the combined view composite result to a display (3112). In some embodiments of the exemplary method (3100), retrieving the MPD may include requesting and receiving the MPD. In some embodiments, the exemplary method described in connection with FIG. 30 and FIG. 31 may be performed by a client device. In some embodiments, the light field content may be three-dimensional light field content. In some embodiments, the exemplary device may include a processor and a non-transient computer-readable medium storing instructions that, when executed by the processor, operate to perform any of the methods described above.

[0316] In some embodiments, generating a view of light field content may include: decoding a frame of light field content; and combining two or more views represented in the frame to generate the generated view, wherein the resulting obtained (or retrieved in some embodiments) light field content includes a frame of light field content. In some embodiments, an exemplary method may include: receiving a media manifest file containing information on a plurality of view subsets of a view of light field content; selecting one of the plurality of view subsets; obtaining light field content associated with the selected view subset; decoding a frame of light field content; combining two or more views represented in the frame to generate a combined view composite result; and rendering the combined view composite result to a display. In some embodiments, a sample method may include: receiving a media manifest file containing information on a plurality of view subsets of a view of light field content; selecting one of the plurality of view subsets; obtaining light field content associated with the selected view subset; generating a view from the obtained light field content; and rendering the generated view to a display. In some embodiments of the sample method, generating one or more views from the acquired light field content may include: decoding a frame of the light field content; and generating a generated view by combining two or more views represented in the frame, so that the acquired light field content includes a frame of the light field content. Some embodiments of the device may include a processor; and a non-transient computer-readable medium storing instructions that, when executed by the processor, operate to perform any one of the exemplary methods listed above.

[0317] FIG. 32 is a flowchart illustrating an exemplary process for a viewing client according to some embodiments. In some embodiments, the exemplary process (3200) may include receiving a media manifest file (3202) that identifies a plurality of representations of a multiview video, and as a result, at least a first representation among the plurality of representations may include a first subsampling of the view and at least a second representation among the plurality of representations may include a second subsampling of the view that is different from the first subsampling of the view. In some embodiments, the exemplary process (3200) may further include selecting a selected representation from the plurality of representations (3204). In some embodiments, the exemplary process (3200) may further include searching for a selected representation (3206). In some embodiments, the exemplary process (3200) may further include rendering a selected representation (3208). In some embodiments, the device may include a processor and a non-transient computer-readable medium that stores instructions that, when executed by the processor, operate to perform an exemplary process (3200).

[0318] Although methods and systems according to some embodiments are discussed in the context of displays, some embodiments may also be applied in the context of virtual reality (VR), mixed reality (MR), and augmented reality (AR). Additionally, although the term “head-mounted display (HMD)” is used herein according to some embodiments, some embodiments may be applied to wearable devices (which may or may not be attached to the head) that are compatible with VR, AR, and / or MR, for example, in some embodiments.

[0319] An exemplary method according to some embodiments may include the following: receiving a media manifest file identifying a plurality of representations of a multiview video—at least one of the plurality of representations having a first subsampling of the view and at least one of the plurality of representations having a second subsampling of the view different from the first subsampling of the view—selecting a selected representation from the plurality of representations; retrieving a selected representation; and rendering the selected representation.

[0320] In some embodiments of the exemplary method, each of the multiple expressions may have a density of views facing each specific direction.

[0321] In some embodiments of the exemplary method, the media manifest file may identify each density of the view and each specific orientation for one or more representations of a plurality of representations.

[0322] In some embodiments of the exemplary method, two or more different subsamplings of a view may differ, at least, in relation to the density of the view of the subsampling of the view facing a particular direction.

[0323] Some embodiments of the exemplary method may further include tracking the user's view direction, wherein selecting the selected representation may include selecting one subsampling of the view among two or more subsamplings of the view, and the selected subsampling of the view has a high density view facing the user's tracked view direction.

[0324] Some embodiments of the exemplary method may further include tracking the user's view direction, wherein selecting a selected representation may include selecting a selected representation based on the user's tracked view direction.

[0325] Some embodiments of the exemplary method may further include obtaining the user's view direction, wherein selecting a selected representation includes selecting a selected representation based on the user's obtained view direction.

[0326] In some embodiments of the exemplary method, the selection of the chosen expression may be based on the user's position.

[0327] In some embodiments of the exemplary method, the selection of the chosen expression may be based on bandwidth constraints.

[0328] In some embodiments of the exemplary method, at least one of the plurality of representations may include a view of higher density with respect to the first view direction than to the second view direction.

[0329] Some embodiments of the exemplary method may further include generating a signal for display using a rendered representation.

[0330] Some embodiments of the exemplary method may further include tracking the user's head position, and the selection of the selected expression may be based on the user's head position.

[0331] Some embodiments of the exemplary method may further include tracking the direction of the user's gaze, and the selection of the selected expression may be based on the direction of the user's gaze.

[0332] Some embodiments of the exemplary method may further include determining the user's viewpoint using the direction of the user's gaze, wherein selecting the selected expression may include selecting the selected expression based on the user's viewpoint.

[0333] Some embodiments of the exemplary method may further include determining the user's viewpoint using the direction of the user's gaze; and selecting at least one subsampling of the view of the multiview video, wherein selecting at least one subsampling of the view may include selecting at least one subsampling of the view within a threshold viewpoint angle of the user's viewpoint.

[0334] Some embodiments of the exemplary method may further include interpolating at least one view, wherein the selected representation may be selected from a plurality of representations and at least one view.

[0335] In some embodiments of the exemplary method, the media manifest file may contain priority data for one or more views, and interpolating at least one view uses the priority data.

[0336] In some embodiments of the exemplary method, the media manifest file may contain priority data for one or more views, and selecting the selected representation uses the priority data.

[0337] Some embodiments of the exemplary method may further include: obtaining light field content associated with a selected representation; decoding a frame of light field content; combining two or more views represented in a frame to generate a combined view composite result; and rendering the combined view composite result to a display.

[0338] In some embodiments of the exemplary method, the frame may include a frame-packed representation of two or more views corresponding to a selected representation.

[0339] In some embodiments of the exemplary method, for at least one of the plurality of representations, the media manifest file may include information corresponding to two or more views of the light field content.

[0340] In some embodiments of the exemplary method, the selection of the selected representation may be based on at least one of the following criteria: user head position, eye tracking, complexity of the content, display capability, and bandwidth available for retrieving light field content.

[0341] In some embodiments of the exemplary method, selecting a selected expression may include: predicting the user's point in time; and selecting a selected expression based on the user's predicted point in time.

[0342] Some embodiments of the exemplary method may further include the following: obtaining light field content associated with a selected representation; generating a generated view of the light field content from the obtained light field content; and rendering the generated view to a display.

[0343] In some embodiments of the exemplary method, generating a generated view of light field content may include: decoding a frame of light field content; and generating a generated view by combining two or more views represented in the frame, wherein the light field content obtained may include a frame of light field content.

[0344] Some embodiments of the exemplary method may further include: decoding a frame of light field content; and combining two or more views represented in the frame to generate a combined view composite result, wherein a plurality of representations of the multiview video may include a plurality of subsamplings of the views of the light field content, and rendering the selected representation may include rendering the combined view composite result to a display.

[0345] Some embodiments of the exemplary method may further include: requesting a media manifest file from a server; and requesting light field content associated with a selected view subset, wherein obtaining light field content associated with a selected view subset may include executing a process selected from the group consisting of: retrieving light field content associated with a selected view subset from a server; requesting light field content associated with a selected view subset from a server; and receiving light field content associated with a selected view subset.

[0346] In some embodiments of the exemplary method, combining two or more views represented in a frame may include using view composition techniques.

[0347] In some embodiments of the exemplary method, selecting one of a plurality of subsamplings of views may also select a view subset based on at least one of the following criteria: user head position, eye tracking, complexity of content, display capability, and bandwidth available for retrieving light field content.

[0348] In some embodiments of the exemplary method, selecting one of a plurality of subsamplings of a view may include: predicting the user's viewpoint; and selecting a subset of views based on the user's predicted viewpoint.

[0349] An exemplary device according to some embodiments may include: a processor; and a non-transient computer-readable medium storing instructions that, when executed by the processor, operate to perform any one of the methods listed above.

[0350] An exemplary method according to some embodiments may include the following: receiving a media manifest file containing information about a plurality of subsampled representations of a view of light field video content; selecting one of the plurality of subsampled representations; obtaining the selected subsampled representation; interpolating one or more interpolated subviews from the selected subsampled representation using information within the manifest file corresponding to each of the one or more views; compositing one or more composite views from one or more interpolated subviews; and displaying one or more composite views.

[0351] Some embodiments of the exemplary method may further include estimating the bandwidth available for streaming light field video content, and as a result, selecting a subsampled representation among a plurality of subsampled representations is based on the estimated bandwidth.

[0352] Some embodiments of the exemplary method may further include tracking the user's position, and as a result, selecting a subsampled representation among a plurality of subsampled representations is based on the user's position.

[0353] Some embodiments of the exemplary method may further include requesting light field video content from a server, wherein obtaining a selected subsampled representation may include executing a process selected from the group consisting of: retrieving a selected subsampled representation from a server, requesting a selected subsampled representation from a server, and receiving a selected subsampled representation.

[0354] In some embodiments of the exemplary method, the information within the manifest file may include location data for two or more views.

[0355] In some embodiments of the exemplary method, the information in the manifest file may include interpolation priority data for one or more of the plurality of views, and the selection of one of the plurality of subsampled representations may be based on the interpolation priority data for one or more of the plurality of views.

[0356] Some embodiments of the exemplary method may further include tracking the user's head position, wherein the selection of one of a plurality of subsampled representations is based on the user's head position.

[0357] Some embodiments of the exemplary method may further include tracking the direction of the user's gaze, wherein the selection of one of a plurality of subsampled representations may be based on the direction of the user's gaze.

[0358] Some embodiments of the exemplary method may further include: determining the user's viewpoint from the direction of the user's gaze; and selecting one or more subviews of light field video content from a group comprising one or more interpolated views and selected subsampled representations, wherein compositing one or more composite views from one or more interpolated subviews may include compositing one or more composite views of the light field using one or more selected subviews and the user's viewpoint.

[0359] Some embodiments of the exemplary method may further include displaying a composite view of the light field.

[0360] In some embodiments of the exemplary method, selecting one or more subviews of the light field may include selecting one or more subviews within a threshold angle of the user's viewpoint.

[0361] Some embodiments of the exemplary method may further include determining the user's viewpoint from the direction of the user's gaze,

[0362] Here, selecting one of multiple subsampled representations may include selecting a subsampled representation based on the user's viewpoint.

[0363] An exemplary device according to some embodiments may include: a processor; and a non-transient computer-readable medium storing instructions that, when executed by the processor, operate to perform any one of the methods listed above.

[0364] An exemplary method according to some embodiments may include: receiving a media manifest file containing information on a plurality of subsamplings of a view of light field content; selecting one of the plurality of subsamplings of the view; acquiring light field content associated with the selected subsampling of the view; decoding a frame of light field content; combining two or more views represented in the frame to generate a combined view composite result; and rendering the combined view composite result to a display, wherein the acquired light field content includes a frame of light field content.

[0365] Some embodiments of the exemplary method may further include: requesting a media manifest file from a server; and requesting light field content associated with a selected view subset, wherein obtaining light field content associated with a selected view subset may include executing a process selected from the group consisting of: retrieving light field content associated with a selected view subset from a server; requesting light field content associated with a selected view subset from a server; and receiving light field content associated with a selected view subset.

[0366] In some embodiments of the exemplary method, the frame may include a frame-packed representation of two or more views corresponding to a selected view subset.

[0367] In some embodiments of the exemplary method, the media manifest file may contain information corresponding to two or more views corresponding to a selected view subset.

[0368] In some embodiments of the exemplary method, for at least one of a plurality of subsamplings of a view, the media manifest file may include information corresponding to two or more views of light field content.

[0369] In some embodiments of the exemplary method, selecting one of a plurality of subsamplings of a view may include parsing information within a media manifest file for a plurality of subsamplings of a view of light field content.

[0370] In some embodiments of the exemplary method, combining two or more views represented in a frame may include using view composition techniques.

[0371] In some embodiments of the exemplary method, selecting one of a plurality of subsamplings of views may also select a view subset based on at least one of the following criteria: eye tracking, complexity of the content, display capability, and bandwidth available for retrieving light field content.

[0372] In some embodiments of the exemplary method, selecting one of a plurality of subsamplings of a view may include: predicting the user's viewpoint; and selecting a subset of views based on the user's predicted viewpoint.

[0373] An exemplary device according to some embodiments may include: a processor; and a non-transient computer-readable medium storing instructions that, when executed by the processor, operate to perform any one of the methods listed above.

[0374] An exemplary method according to some embodiments may include the following: receiving a media manifest file containing information on a plurality of subsamplings of a view of light field content; selecting one of the plurality of subsamplings of the view; obtaining light field content associated with the selected view subset; generating a view from the obtained light field content; and rendering the generated view to a display.

[0375] In some embodiments of the exemplary method, generating a view from the acquired light field content may include interpolating the view from the light field content associated with the selected view subset using information within the manifest file corresponding to each view in order to generate the generated view.

[0376] In some embodiments of the exemplary method, generating one or more views from the acquired light field content may include: decoding a frame of the light field content; and generating a generated view by combining two or more views represented in the frame, wherein the acquired light field content may include a frame of the light field content.

[0377] An exemplary device according to some embodiments may include: a processor; and a non-transient computer-readable medium storing instructions that, when executed by the processor, operate to perform any one of the methods listed above.

[0378] An exemplary method according to some embodiments may include the following: receiving a media manifest file identifying multiple subsamplings of a view of a multiview video—wherein multiple subsamplings of a view include two or more different densities of a view—selecting a selected subsampling from the multiple subsamplings of a view; retrieving the selected subsampling; and rendering the selected subsampling.

[0379] An exemplary device according to some embodiments may include: a processor; and a non-transient computer-readable medium storing instructions that, when executed by the processor, operate to perform any one of the methods listed above.

[0380] An exemplary method according to some embodiments may include: rendering a representation of a view comprising an entire array of light field video content; transmitting the rendered entire array representation of the view; obtaining the client's current viewpoint; predicting a future viewpoint using the current viewpoint and a viewpoint motion model; prioritizing a plurality of subsampled representations of the view of light field video content; rendering a plurality of prioritized subsampled representations of the view of light field video content; and transmitting a plurality of prioritized subsampled representations of the view.

[0381] An exemplary device according to some embodiments may include: a processor; and a non-transient computer-readable medium storing instructions that, when executed by the processor, operate to perform any one of the methods listed above.

[0382] An exemplary method according to some embodiments may include the following: selecting a plurality of sub-views of light field video content; generating streaming data for each of the plurality of sub-views of light field video content; and generating a media manifest tile containing streaming data for each of the plurality of sub-views of light field video content.

[0383] An exemplary device according to some embodiments may include: a processor; and a non-transient computer-readable medium storing instructions that, when executed by the processor, operate to perform any one of the methods listed above.

[0384] An exemplary method according to some embodiments may include the following: receiving a request for information about light field video content; transmitting a media manifest file containing information about a plurality of subsampled representations of a view of light field video content if the request is a new session request; and transmitting a data segment containing a subset of light field video content if the request is a subset data segment request.

[0385] An exemplary device according to some embodiments may include: a processor; and a non-transient computer-readable medium storing instructions that, when executed by the processor, operate to perform any one of the methods listed above.

[0386] An exemplary signal according to some embodiments may include a signal conveying a representation of a view comprising a whole array of light field video content and a plurality of subsampled representations of a view of the light field video content.

[0387] An exemplary signal according to some embodiments may include a signal that conveys a plurality of sub-views of light field video content.

[0388] An exemplary signal according to some embodiments may include a signal that transmits streaming data for each of a plurality of sub-views of light field video content.

[0389] An exemplary signal according to some embodiments may include a signal that conveys information about a plurality of subsampled representations of a view of light field video content.

[0390] An exemplary signal according to some embodiments may include a signal that conveys a data segment containing a subset of light field video content.

[0391] It should be noted that various hardware elements of one or more of the described embodiments are referred to as “modules” that carry out (i.e., perform, execute, and etc.) the various functions described herein in relation to each module. As used herein, a module includes hardware that is considered appropriate by a person skilled in the relevant art for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices). It should also be noted that each described module may include executable instructions to perform one or more functions described as being performed by each module, and those instructions may take or include the form of hardware (i.e. embedded in hardware) instructions, firmware instructions, software instructions, and / or such, and may be stored on any suitable non-transient computer-readable medium or medium, such as those generally referred to as RAM, ROM, etc.

[0392] Although features and elements are described above in specific combinations, those skilled in the art will recognize that each feature or element may be used alone or in any combination with other features and elements. Additionally, the methods described herein may be implemented as computer programs, software, or firmware integrated into a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, magnetic media such as read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, embedded hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). A processor associated with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

Claim 1 A method implemented by a server comprises: generating multiple representations of light field video content by selecting a quantity of subviews of an integral light field image in various subsamplings to configure multiple light field data subsets, wherein compiling the subviews into each subset is performed based on the light field video content; receiving a message from a device indicating a request for a media manifest file; transmitting the media manifest file identifying multiple representations of the light field video content to the device, wherein at least one first representation among the multiple representations includes a first subview subset and at least two representations among the multiple representations include a different second subview subset; receiving a request from the device for a representation selected from the multiple representations; and transmitting the selected representation to the device. Claim 2 A method implemented by a server in which, in claim 1, each of the plurality of expressions has a density of views facing a specific respective direction. Claim 3 A method implemented by a server, wherein the media manifest file identifies the density of each of the views and each of the specific directions for one or more of the plurality of representations. Claim 4 A method implemented by a server according to claim 1, wherein the first and second sub-view subsets are at least different in relation to the density of views of the sub-view subsets facing specific directions. Claim 5 A method implemented by a server in which the selected expression in paragraph 1 is based on the user's position. Claim 6 A method implemented by a server according to claim 1, wherein at least one of the plurality of expressions includes views of higher density with respect to the first view direction than to the second view direction. Claim 7 A method implemented by a server in claim 1, wherein the media manifest file includes priority data for one or more views, and the selected representation is based on the priority data. Claim 8 A server comprising: a processor; and a non-transient computer-readable medium storing instructions, wherein the instructions, when executed by the processor, cause the server to generate multiple representations of light field video content by selecting a quantity of subviews of an integral light field image in various subsamplings to configure multiple light field data subsets—compiling the subviews into each subset is performed based on the light field video content—receive a message indicating a request for a media manifest file from a device, and transmit the media manifest file identifying the multiple representations of the light field video content to the device—wherein at least a first representation among the multiple representations includes a first subview subset and at least a second representation among the multiple representations includes a different second subview subset—; receive a request from the device for a representation selected from the multiple representations; and operate to cause the selected representation to be transmitted to the device. Claim 9 In paragraph 8, a server in which each of the above multiple expressions has a density of views facing a specific respective direction. Claim 10 In claim 9, the media manifest file is a server that identifies the density of each of the views and each of the specific directions for one or more of the plurality of representations. Claim 11 In paragraph 8, the server, wherein the first and second sub-view subsets are at least different in relation to the density of views of the sub-view subsets facing specific directions. Claim 12 In paragraph 8, the server, wherein the selected expression is based on the user's position. Claim 13 A server according to claim 8, wherein at least one of the plurality of expressions includes views of higher density for the first view direction than for the second view direction. Claim 14 In paragraph 8, the media manifest file contains priority data for one or more views, and the selected representation is based on the priority data, the server.

Images