Systems and methods for adaptive microlens light field transport and rendering

By selecting sub-sampling microlenses and interpolating and reconstructing light field video content based on factors such as user viewpoint and bandwidth, the problem of high-fidelity light field video data compression and transmission is solved, achieving efficient light field video content transmission and visualization.

CN114375583BActive Publication Date: 2026-11-10INTERDIGITAL VC HOLDINGS INC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202080063308.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-07-23
Filing Date
2020-07-20
Publication Date
2026-11-10
Estimated Expiration
2040-07-20

AI Technical Summary

Technical Problem

Existing technologies struggle to efficiently compress and transmit high-fidelity light field video data, leading to difficulties in real-time transmission and visualization.

Method used

By selecting sub-sampling microlenses, the light field video content is interpolated and reconstructed based on factors such as user viewpoint, bandwidth, display capability, and gaze direction, thereby optimizing the light field spatial resolution to improve view performance.

Benefits of technology

It achieves efficient transmission and visualization of light field video content, adapts to different bandwidths and display capabilities, and improves view quality and transmission efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114375583B_ABST
    Figure CN114375583B_ABST
Patent Text Reader

Abstract

Some embodiments of the method may include: streaming a microlens representation of the light field video content; and changing the resolution of the microlens representation. Some embodiments of the method may include: selecting a microlens representation from a plurality of microlens representations of a portion of the light field content; retrieving a subsampled microlens representation of the selected microlens representation; and interpolating a view from the subsampled microlens representation.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Cross-references to related applications

[0002] This application is a non-provisional application filed on July 23, 2019, entitled “SYSTEM AND METHOD FOR ADAPTIVELENSLET LIGHT FIELD TRANSMISSION AND RENDERING”, and claims the benefit of it pursuant to 35 USC §119(e), which is incorporated herein by reference in its entirety. Background Technology

[0003] High-fidelity light fields, used to represent 3D scenes, can contain large amounts of data. To support real-time transmission and visualization, efficient data distribution optimization methods can be used. Various lossless and lossy bitrate reduction and compression methods have been developed to compress traditional 2D video. Summary of the Invention

[0004] Example methods according to some embodiments may include: receiving from a server a media manifest file containing a plurality of sub-sampled lenslet representations describing portions of a light field video content; selecting a sub-sampled lenslet representation from the plurality of sub-sampled lenslet representations; retrieving the selected sub-sampled lenslet representation from the server; interpolating a view from the retrieved selected sub-sampled lenslet representation using the description of the selected sub-sampled lenslet representation in the manifest file; and displaying the interpolated view.

[0005] Some embodiments of the example method may also include: determining an estimated bandwidth that can be used for streaming the light field video content.

[0006] For some embodiments of the example method, the selection of the subsampled microlens representation may be based on at least one of the following: the user's viewpoint, estimated bandwidth, or the display capability of the viewing client.

[0007] Some embodiments of the example method may further include: predicting the user's predicted viewpoint such that the user's viewpoint is the user's predicted viewpoint.

[0008] For some embodiments of the example method, selecting the subsampled microlens representation may include: determining a corresponding minimum support bandwidth of at least one of the plurality of subsampled microlens representations; and selecting the subsampled microlens representation having the largest minimum support bandwidth among a plurality of corresponding minimum support bandwidths that are less than the estimated bandwidth.

[0009] Some embodiments of the example method may further include: determining the estimated maximum content size supported by the estimated bandwidth, such that selecting the subsampled microlens representation can select one of the plurality of subsampled microlens representations having a content size smaller than the estimated maximum content size.

[0010] Some embodiments of the example method may also include: tracking the user's gaze direction such that selecting the sub-sampling microlens represents using the user's gaze direction.

[0011] For some embodiments of the example method, selecting the subsampled microlens representation may include: selecting a subsampled microlens representation with a density higher than a density threshold for a portion of the light field content located within a gaze threshold in the user's gaze direction.

[0012] Some embodiments of the example method may also include: predicting the user's viewpoint; and adjusting the selected microlens representation for the predicted viewpoint.

[0013] Some embodiments of the example method may further include: selecting a light field spatial resolution; dividing the light field content into portions corresponding to the light field spatial resolution; and selecting a microlens image for at least one frame of at least one subsampled microlens representation of at least a portion of the light field content, such that selecting the subsampled microlens representation may select a corresponding subsampled microlens representation for at least a portion of the light field content, and such that the corresponding microlens image can be used for interpolated views from the subsampled microlens representation.

[0014] Some embodiments of the example method may also include: adjusting the spatial resolution of the light field to improve the performance metrics of the interpolated view.

[0015] For some embodiments of the example method, interpolating the view from the retrieved subsampled microlens representation may include: unpacking the retrieved subsampled microlens representation to the original microlens position of the portion of the light field video content indicated in the manifest file; and interpolating microlens samples omitted from the retrieved subsampled microlens representation.

[0016] In some embodiments of the example method, the interpolated view representing the retrieved subsampled small lens can generate a complete light field region image of the portion of the light field video content.

[0017] For some embodiments of the example method, the subsampling microlens representation may be selected based on at least one of the following: the density of the selected subsampling microlens representation, or the range of the selected subsampling microlens representation.

[0018] Example apparatuses according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, operate to perform the method of any of the above-listed claims.

[0019] Example methods according to some embodiments may include: selecting a microlens representation from a plurality of microlens representations of a portion of light field content described in a media manifest file; retrieving a subsampled microlens representation of the selected microlens representation from a server; and interpolating a view from the subsampled microlens representation.

[0020] Some embodiments of the example method may also include: retrieving a media manifest file representing a portion of the light field video content using multiple small lenses; and displaying the interpolated view.

[0021] For some embodiments of the example method, the interpolated view from the retrieved subsampled microlens representation can use the description of the microlens representation in the manifest file.

[0022] Some embodiments of the example method may further include: determining an estimated bandwidth between the client and the server, such that selecting the small lens indicates that the estimated bandwidth can be used.

[0023] For some embodiments of the example method, the description of at least one of the plurality of microlens representations may include information about at least one of the range or density of the respective microlens representation.

[0024] For some embodiments of the example method, selecting the subsampled microlens representation may include: selecting a subsampled microlens representation with the largest range.

[0025] For some embodiments of the example method, selecting the subsampled microlens representation may include: selecting the subsampled microlens representation with the highest density.

[0026] In some embodiments of the example method, selecting the small lens indicates that the client's capabilities can be used.

[0027] For some embodiments of the example method, selecting the subsampled microlens representation may include: selecting a subsampled microlens representation with a density higher than a threshold supported by the client.

[0028] In some embodiments of the example method, the capability of the client may be the maximum microlens density supported by the client.

[0029] For some embodiments of the example method, the interpolated view from the subsampled microlens representation can use the description of the microlens representation in the manifest file.

[0030] Some embodiments of the example method may also include: updating the user's viewpoint; and adjusting the selected microlens representation for the updated viewpoint.

[0031] Some embodiments of the example method may also include: predicting the user's viewpoint, such that selecting a subsampled microlens representation may include: selecting a subsampled microlens representation with a density above a threshold for a portion of the light field content that lies within a threshold of the predicted viewpoint.

[0032] Some embodiments of the example method may also include: selecting a subsampling rate for the selected microlens representation.

[0033] Some embodiments of the example method may further include: estimating the bandwidth available for streaming optical field video content, such that the sub-sampling rate can be selected using the estimated available bandwidth.

[0034] Some embodiments of the example method may further include: for each frame of each subsampled microlens representation of each part of the light field content, selecting a microlens image such that the microlens representation is selected from the plurality of microlens representations; selecting a corresponding subsampled microlens representation for each part of the light field content such that the interpolated view from the subsampled microlens representation uses the corresponding microlens image; and selecting the microlens image from the plurality of microlens images based on the estimated quality of the interpolation result.

[0035] Some embodiments of the example method may further include: determining the corresponding interpolation result estimation quality of the plurality of microlens images, such that the microlens image is selected based on which of the plurality of microlens images has a determined highest corresponding interpolation result estimation quality.

[0036] Example apparatuses according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operable, when executed by the processor, to perform any of the methods listed above.

[0037] Example methods according to some embodiments may include: streaming light field microlens representations of light field video content; and changing the resolution of the light field microlens representation.

[0038] Example apparatuses according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operable, when executed by the processor, to perform any of the methods listed above.

[0039] Example methods according to some embodiments may include: selecting a microlens representation from a plurality of microlens representations of a portion of the light field content; retrieving a subsampled microlens representation of the selected microlens representation; and interpolating a view from the subsampled microlens representation to reconstruct missing microlens samples in the subsampled representation.

[0040] Example apparatuses according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operable, when executed by the processor, to perform any of the methods listed above.

[0041] Example methods according to some embodiments may include: retrieving a subsampled microlens representation of light field content; and reconstructing microlens samples omitted from the subsampled microlens representation by interpolating the retrieved subsampled microlens representation.

[0042] Example apparatuses according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operable, when executed by the processor, to perform any of the methods listed above.

[0043] Example methods according to some embodiments may include: sending a media manifest file of a plurality of subsampled microlens representations describing portions of a light field video content; receiving information indicating a subsampled microlens representation selected from the plurality of subsampled microlens representations; and sending the selected subsampled microlens representation.

[0044] Example apparatuses according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operable, when executed by the processor, to perform any of the methods listed above.

[0045] Example apparatuses according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, operate to: transmit a media manifest file of a plurality of subsampled microlens representations describing portions of a light field video content; receive information indicating a subsampled microlens representation selected from the plurality of subsampled microlens representations; and transmit the selected subsampled microlens representation. Attached Figure Description

[0046] Figure 1A This is a system diagram of an example communication system according to some embodiments.

[0047] Figure 1B According to some embodiments, it is possible to Figure 1A The diagram shows a system diagram of an example wireless transmit / receive unit (WTRU) used in a communication system.

[0048] Figure 2 This is a schematic diagram illustrating an example full light field rendered using a 5×5 subview according to some embodiments.

[0049] Figure 3A This is a schematic diagram illustrating an example small lens representation according to some embodiments.

[0050] Figure 3B This is a schematic diagram illustrating an exemplary view array representation according to some embodiments.

[0051] Figures 4A-4B This is a schematic diagram illustrating an example of light field data in a small lens format according to some embodiments.

[0052] Figure 5 This is a system diagram illustrating a set of example interfaces for a viewing client according to some embodiments.

[0053] Figure 6 This is a message sequence diagram illustrating an example process of a typical session sequence according to some embodiments.

[0054] Figures 7A-7B This is a flowchart illustrating an example process of a content server process according to some embodiments.

[0055] Figure 8 This is a schematic diagram illustrating an exemplary fulllenslet light field divided into multiple regions according to some embodiments.

[0056] Figures 9A-9C This is a schematic diagram illustrating an exemplary set of subsamples with varying subsample densities at a first time step, according to some embodiments.

[0057] Figures 10A-10C This is a schematic diagram illustrating an example subsample set with varying subsample densities at a second time step, according to some embodiments.

[0058] Figure 11A-11C This is a schematic diagram illustrating example subviews selected according to some embodiments for being packaged into a set of subsamples of a dense integral image.

[0059] Figure 12 This is a data structure diagram illustrating an example MPEG-DASH Media Presentation Description (MPD) according to some embodiments.

[0060] Figure 13 This is a data structure diagram illustrating an example media presentation description (MPD) with example microlens light field description (one or more), subsample set, resolution, and bit rate according to some embodiments.

[0061] Figure 14 This is a process diagram illustrating an example small lens array reconstruction process according to some embodiments.

[0062] Figure 15 This is a flowchart illustrating an example process for a viewing client according to some embodiments.

[0063] Figure 16A This is a schematic diagram illustrating an example multi-view array representation according to some embodiments.

[0064] Figure 16B This is a schematic diagram illustrating an exemplary total microlens representation according to some embodiments.

[0065] Figure 17A This is a schematic diagram illustrating an example multi-view array representation of the illustrated ROI according to some embodiments.

[0066] Figure 17B This is a schematic diagram illustrating an exemplary total microlens representation having the illustrated ROI according to some embodiments.

[0067] Figure 18A This is a schematic diagram illustrating an example multi-view array representation with the illustrated ROI and selected views according to some embodiments.

[0068] Figure 18B This is a schematic diagram illustrating an exemplary total microlens representation having the illustrated ROI according to some embodiments.

[0069] Figure 19 This is a process diagram illustrating an example subsampling process for selecting a small lens view according to some embodiments.

[0070] Figure 20 This is an image illustration showing an example full microlens image with 425×425 samples and 5×5 views according to some embodiments.

[0071] Figure 21 This is an image illustration showing an example subsampled microlens image with 255×255 samples and 3×3 views according to some embodiments.

[0072] Figure 22 This is an image illustration showing an example subsampled microlens image with 255×85 samples and 1×3 views according to some embodiments.

[0073] Figure 23 This is an image illustration showing an example subsampled microlens image with 85×85 samples and 1×1 views according to some embodiments.

[0074] Figures 24A-24C This is a schematic diagram illustrating an example microlens light field subsampled with two subsampling densities according to some embodiments.

[0075] Figure 25 This is a message sequence diagram illustrating an example process for adaptive optical field streaming using estimated bandwidth and view interpolation, according to some embodiments.

[0076] Figure 26 This is a flowchart illustrating an example process for a viewing client according to some embodiments.

[0077] Figure 27 This is a flowchart illustrating an example process for a viewing client according to some embodiments.

[0078] The entities, connections, arrangements, etc., depicted and described in connection with the various figures are given as examples rather than as limitations. Therefore, any and all statements or other indications regarding what a particular figure “depicts,” what a particular element or entity in a particular figure “is” or “has,” and any and all similar statements—which may be isolated and interpreted as absolute and therefore limiting outside the context—may be properly interpreted only if they are preceded by a constructive prefix such as “in at least one embodiment, …”. For the sake of brevity and clarity, this implied prefix is ​​not repeated in the detailed description. Detailed Implementation

[0079] In some embodiments described herein, the wireless transmit / receive unit (WTRU) can be used as, for example, a viewing client, a content server, a sensor, or a display.

[0080] Figure 1AThis diagram illustrates an example communication system 100 that can implement one or more of the disclosed embodiments. The communication system 100 can be a multiple access system providing voice, data, video, messaging, broadcasting, and other content to multiple wireless users. The communication system 100 enables multiple wireless users to access such content by sharing system resources, including wireless bandwidth. For example, the communication system 100 can use one or more channel access methods, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), Single Carrier FDMA (SC-FDMA), Zero-Tail Unique Word DFT-Extended OFDM (ZT UW DTS-s OFDM), Unique Word OFDM (UW-OFDM), Resource Block Filtering OFDM, and Filter Bank Multicarrier (FBMC), etc.

[0081] like Figure 1A As shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106, public switched telephone network (PSTN) 108, Internet 110, and other networks 112. However, it should be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network components. Each WTRU 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. For example, any WTRU 102a, 102b, 102c, or 102d may be referred to as a “station” and / or “STA”, and may be configured to transmit and / or receive wireless signals. It may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain environments), consumer electronics, and devices operating on commercial and / or industrial wireless networks, etc. Any of WTRU 102a, 102b, 102c, or 102d may be interchangeably referred to as a UE.

[0082] The communication system 100 may also include base stations 114a and / or 114b. Each base station 114a, 114b may be any type of device configured to enable its access to one or more communication networks (e.g., CN 106, Internet 110, and / or other networks 112) by wirelessly interfacing with at least one of WTRUs 102a, 102b, 102c, 102d. For example, base stations 114a, 114b may be base transceiver stations (BTS), node B, e-node B, home node B, home e-node B, gNB, NR node B, site controller, access point (AP), and wireless routers, etc. Although each base station 114a, 114b is described as a single component, it should be understood that base stations 114a, 114b may include any number of interconnected base stations and / or network components.

[0083] Base station 114a may be part of RAN 104 / 113, and the RAN may also include other base stations and / or network components (not shown), such as base station controllers (BSCs), radio network controllers (RNCs), relay nodes, etc. Base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies called cells (not shown). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide radio service coverage for a specific geographic area that is relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, a cell associated with base station 114a may be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, that is, each transceiver corresponds to one sector of the cell. In embodiments, base station 114a may use multiple-input multiple-output (MIMO) technology and may use multiple transceivers for each sector of the cell. For example, by using beamforming, signals can be transmitted and / or received in a desired spatial direction.

[0084] Base stations 114a and 114b can communicate with one or more of WTRUs 102a, 102b, 102c, and 102d via air interface 116, wherein the air interface can be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). Air interface 116 can be established using any suitable radio access technology (RAT).

[0085] More specifically, as described above, the communication system 100 can be a multiple access system and can use one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, and SC-FDMA, etc. For example, base station 114a in RAN 104 / 113 and WTRUs 102a, 102b, and 102c can implement a certain radio technology, such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), wherein the technology can use Wideband CDMA (WCDMA) to establish the air interface 116. WCDMA may include communication protocols such as High-Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High-Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High-Speed ​​UL Packet Access (HSUPA).

[0086] In an embodiment, base station 114a and WTRUs 102a, 102b, 102c may implement a certain radio technology, such as Evolved UMTS Terrestrial Radio Access (E-UTRA), wherein the technology may use Long Term Evolution (LTE) and / or Advanced LTE (LTE-A) and / or Advanced LTA Pro (LTE-A Pro) to establish air interface 116.

[0087] In an embodiment, base station 114a and WTRUs 102a, 102b, 102c may implement a certain radio technology, such as NR radio access, wherein the radio technology may use a novel radio (NR) to establish air interface 116.

[0088] In this embodiment, base station 114a and WTRUs 102a, 102b, and 102c can implement various radio access technologies. For example, base station 114a and WTRUs 102a, 102b, and 102c can jointly implement LTE radio access and NR radio access (e.g., using the dual connectivity (DC) principle). Therefore, the air interface used by WTRUs 102a, 102b, and 102c can be characterized by various types of radio access technologies and / or transmissions sent to / from various types of base stations (e.g., eNBs and gNBs).

[0089] In other embodiments, base station 114a and WTRUs 102a, 102b, 102c may implement the following radio technologies, such as IEEE 802.11 (i.e., WiFi), IEEE 802.16 (Global Microwave Access Interoperability (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Provisional Standard 2000 (IS-2000), Provisional Standard 95 (IS-95), Provisional Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rate for GSM Evolution (EDGE), and GSM EDGE (GERAN), etc.

[0090] Figure 1A Base station 114b can be a wireless router, home node B, home e node B, or access point, and can use any suitable RAT to facilitate wireless connectivity in a local area, such as a business premises, residence, vehicle, campus, industrial facility, air corridor (e.g., for use by drones), and road, etc. In one embodiment, base station 114b and WTRUs 102c, 102d can establish a wireless local area network (WLAN) by implementing radio technology such as IEEE 802.11. In another embodiment, base station 114b and WTRUs 102c, 102d can establish a wireless personal area network (WPAN) by implementing radio technology such as IEEE 802.15. In yet another embodiment, base station 114b and WTRUs 102c, 102d can establish a picocell or femtocell by using a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.). Figure 1A As shown, base station 114b can be directly connected to the Internet 110. Therefore, base station 114b does not need to access the Internet 110 via CN 106.

[0091] RAN 104 / 113 can communicate with CN 106, which can be any type of network configured to provide voice, data, application, and / or Voice over Internet Protocol (VoIP) services to one or more WTRUs 102a, 102b, 102c, 102d. The data can have different Quality of Service (QoS) requirements, such as different throughput requirements, latency requirements, fault tolerance requirements, reliability requirements, data throughput requirements, and mobility requirements, etc. CN 106 can provide call control, billing services, location-based services, prepaid calling, Internet connectivity, video distribution, etc., and / or can perform advanced security functions such as user authentication. While in Figure 1AWhile not shown, it should be understood that RAN 104 / 113 and / or CN 106 can communicate directly or indirectly with other RANs that use the same RAT or a different RAT as RAN 104 / 113. For example, in addition to connecting with RAN 104 / 113 using NR radio technology, CN 106 can also communicate with other RANs (not shown) using GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technologies.

[0092] CN 106 can also act as a gateway for WTRUs 102a, 102b, 102c, and 102d to access PSTN 108, the Internet 110, and / or other networks 112. PSTN 108 may include a circuit-switched telephone network providing Simple Old-Style Telephone Service (POTS). The Internet 110 may include a globally interconnected computer network equipment system using common communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) from the TCP / IP Internet Protocol suite. Network 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, the other network 112 may include another CN connected to one or more RANs, wherein the one or more RANs may use the same RAT or a different RAT as RAN 104 / 113.

[0093] Some or all of the WTRUs 102a, 102b, 102c, and 102d in the communication system 100 may include multi-mode capability (e.g., WTRUs 102a, 102b, 102c, and 102d may include multiple transceivers communicating with different wireless networks on different wireless links). For example... Figure 1A The WTRU 102c shown can be configured to communicate with base station 114a, which can use cellular-based radio technology, and with base station 114b, which can use IEEE 802 radio technology.

[0094] Figure 1B This is a system diagram illustrating an example of WTRU 102. (See diagram below.) Figure 1B As shown, WTRU 102 may include a processor 118, a transceiver 120, a transmit / receive unit 122, a speaker / microphone 124, a keyboard 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power supply 134, a global positioning system (GPS) chipset 136, and other peripheral devices 138. It should be understood that, while remaining consistent with the embodiments, WTRU 102 may also include any sub-combination of the foregoing components.

[0095] Processor 118 can be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), and a state machine, etc. Processor 118 can perform signal encoding, data processing, power control, input / output processing, and / or any other function that enables WTRU 102 to operate in a wireless environment. Processor 118 can be coupled to transceiver 120, and transceiver 120 can be coupled to transmitting / receiving unit 122. Although Figure 1B While the processor 118 and transceiver 120 are described as separate components, it should be understood that the processor 118 and transceiver 120 can also be integrated into a single electronic component or chip.

[0096] Transmit / receive component 122 may be configured to transmit or receive signals to or from a base station (e.g., base station 114a) via air interface 116. For example, in one embodiment, transmit / receive component 122 may be an antenna configured to transmit and / or receive RF signals. As an example, in an embodiment, transmit / receive component 122 may be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals. In an embodiment, transmit / receive component 122 may be configured to transmit and / or receive RF and optical signals. It should be understood that transmit / receive component 122 may be configured to transmit and / or receive any combination of wireless signals.

[0097] Although Figure 1B The transmit / receive component 122 is described as a single component, but the WTRU 102 may include any number of transmit / receive components 122. More specifically, the WTRU 102 may use MIMO technology. Thus, in an embodiment, the WTRU 102 may include two or more transmit / receive components 122 (e.g., multiple antennas) that transmit and receive radio signals via the air interface 116.

[0098] Transceiver 120 can be configured to modulate signals to be transmitted by transmitter / receiver 122 and demodulate signals received by transmitter / receiver 122. As described above, WTRU 102 can have multimode capability. Therefore, transceiver 120 can include multiple transceivers that allow WTRU 102 to communicate using various RATs (e.g., NR and IEEE 802.11).

[0099] The processor 118 of WTRU 102 can be coupled to a speaker / microphone 124, a keyboard 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) unit or an organic light-emitting diode (OLED) display unit) and can receive user input data from these components. The processor 118 can also output user data to the speaker / microphone 124, keyboard 126, and / or display / touchpad 128. Furthermore, the processor 118 can access and store information from any suitable memory, such as non-removable memory 130 and / or removable memory 132. Non-removable memory 130 can include random access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. Removable memory 132 can include a subscriber identification module (SIM) card, a memory stick, a secure digital card (SD) memory card, etc. In other embodiments, the processor 118 can access and store information from memory that is not actually located in WTRU 102; for example, such memory could be located in a server or home computer (not shown).

[0100] The processor 118 can receive power from the power supply 134 and can be configured to distribute and / or control power for other components in the WTRU 102. The power supply 134 can be any suitable device for powering the WTRU 102. For example, the power supply 134 may include one or more dry cell battery packs (such as nickel-cadmium (Ni-Cd), nickel-zinc (Ni-Zn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, and fuel cells, etc.

[0101] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) related to the current location of the WTRU 102. As a supplement or replacement to the information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) via the air interface 116, and / or determine its location based on signal timing received from two or more nearby base stations. It should be understood that, while remaining consistent with the embodiments, the WTRU 102 may acquire location information using any suitable positioning method.

[0102] The processor 118 can also be coupled to other peripheral devices 138, which may include one or more software and / or hardware modules providing additional features, functions, and / or wired or wireless connectivity. For example, peripheral devices 138 may include accelerometers, electronic compasses, satellite transceivers, digital cameras (for photos and / or video), Universal Serial Bus (USB) ports, vibration devices, television transceivers, hands-free headsets, etc. Modules, FM radio units, digital music players, media players, video game console modules, internet browsers, virtual reality and / or augmented reality (VR / AR) devices, and activity trackers, etc. Peripheral devices 138 may include one or more sensors, which may be one or more of the following: gyroscopes, accelerometers, Hall effect sensors, magnetometers, orientation sensors, proximity sensors, temperature sensors, time sensors, geolocation sensors, altimeters, light sensors, touch sensors, magnetometers, barometers, gesture sensors, biometric sensors, and / or humidity sensors.

[0103] WTRU 102 may include a full-duplex wireless device, wherein the reception or transmission of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous for the wireless device. The full-duplex wireless device may include an interference management unit that reduces and / or substantially eliminates self-interference by means of hardware (e.g., choke coils) or by means of a processor (e.g., a separate processor (not shown) or by means of processor 118) for signal processing. In embodiments, WTRU 102 may include a half-duplex wireless device that transmits and receives some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) or downlink (e.g., for reception).

[0104] Given Figure 1A-1B and Figure 1A-1B The corresponding descriptions may be performed by one or more emulation devices (not shown) that perform one or more or all of the functions described herein with respect to: WTRU 102a-d, base stations 114a-b, and / or any other devices (one or more) described herein. The emulation device may be one or more devices configured to emulate one or more or all of the functions described herein. For example, the emulation device may be used to test other devices and / or simulate network and / or WTRU functions.

[0105] The simulation device may be designed to perform one or more tests on other devices in a laboratory environment and / or a carrier network environment. For example, the one or more simulation devices may perform one or more or all functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices within the communication network. The one or more simulation devices may perform one or more or all functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. The simulation device may be directly coupled to another device for testing purposes and / or may use over-the-air wireless communication to perform tests.

[0106] The one or more simulation devices may perform one or more functions (including all) when implemented / deployed not as part of a wired and / or wireless communication network. For example, the simulation devices may be used in test scenarios in test labs and / or non-deployed (e.g., testing) wired and / or wireless communication networks to test one or more components. The one or more simulation devices may be test equipment. The simulation devices may transmit and / or receive data using direct RF coupling and / or wireless communication via RF circuitry (e.g., which may include one or more antennas).

[0107] Besides spatiotemporal compression methods, another class of bit rate reduction methods transmit portions of information that are multiplexed in time. For CRT displays, multiplexing is widely used by interlacing image lines in analog TV transmission. Another class of compression algorithms uses various prediction methods, typically performed on both the transmission side (encoder or server) and the reception side (decoder). These predictions can be spatial (intra-frame) or temporal (inter-frame).

[0108] Some of these algorithms have been used for light fields. For example, the following paper is understood to describe how the subjective quality of light field rendering is affected by quality switching and stopping (frame freezing) methods and the balance between related trade-offs between transmission bit rate, light field angular resolution and spatial resolution: Kara, Peter A. et al., Evaluation of the Concept of Dynamic Adaptive Streaming of Light Field Video, 64(2) IEEE Transactions on Broadcasting 407-421 (2018).

[0109] Typically, methods for applying predictive coding to the real-time transmission of light fields remain scarce. Examples of these methods are discussed in the following paper on light field compression: Ebrahimi, Touradj et al., JPEG Pleno: Toward an Efficient Representation of Visual Reality, 23(4) IEEE Multimedia 14-20 (October-December 2016) (“Ebrahimi”). This proposal is understood to describe how existing multi-view coding methods (e.g., MPEG HEVC or 3D HEVC) can be used for light field compression. 3D HEVC is an extension of HEVC for supporting depth images.

[0110] Much of the work done in light field compression focuses on multi-view image array-type light field formats, where the light field data consists of multiple views of the entire scene captured from varying viewpoints. In existing multi-view coding standards, even when only a specific subview is viewed, all views typically need to be decoded, as understood in the following article: Sullivan, Gary J et al., Standardized Extensions of High Efficiency Video Coding (HEVC), 7(6) IEEE J. Selected Topics in Signal Proc. (December 2013). Consequently, due to the data dependencies between subviews, it may be impossible to decode only one subview at a time or to decode multiple subviews in parallel.

[0111] Figure 2 This is a schematic diagram illustrating an example full light field rendered using a 5x5 subview according to some embodiments. Figure 2 In this process, a 5×5 virtual camera array of 200 is used to render the full light field. Figure 2 An example multi-view light field with a 5×5 full-view array 202 is shown.

[0112] Figure 3A This is a schematic diagram illustrating an example small lens representation according to some embodiments. Figure 3B This is a schematic diagram illustrating an exemplary array view according to some embodiments. Figure 3A An example of a total small lens light field representation 300 and a region 302 that is part of the total small lens light field representation 300 is shown. Figure 3A Example region 302 in Figure 3B It is shown in more detail below. Figure 3B Example subviews 352 corresponding to example areas 302, 350 are shown. For some embodiments, for example... Figure 3B The example shown is a subview 352 of a scene taken at the same time from different viewpoints.

[0113] Display technologies are emerging that enable free-viewpoint viewing of video content. These displays use light fields that generate massive amounts of data. The rendering and transmission of this light field data can become a bottleneck. Data optimization can be used to avoid this bottleneck. Resolution and frame rate are two aspects of video sequences that can be adapted to enable content streaming via DASH.

[0114] Many adaptive video streaming devices do not utilize the angular aspect of the light field content. One potential problem is how to adapt the view for streaming video content represented as interlaced "microlens" images. Spatially tiled streaming extracts all viewing directions of a subset of the content rather than all viewing directions of a subset of the entire content view, such that, in order to obtain a single view, the tile carries the entire microlens representation.

[0115] Figures 4A-4B This is a schematic diagram illustrating an example of light field data in a small lens format according to some embodiments. Light fields captured by a camera array produce light field data in a multi-view format, such as a full-image array of a scene captured from varying viewpoints. See also... Figure 2 In addition to camera arrays that produce high-resolution subviews, an alternative approach to representing light field data rather than a full-view array is, for example, to produce a much larger number of subviews, each characterized only by a small image region, but with higher angular resolution. Figures 4A-4B Examples of such small lens light field images at 400 and 450 can be seen in the image. Figure 4A A light field region 402 with a set of subviews 404 is shown. Typically, these microlens-type light fields are generated using microlens array optics in a capture device. Similar lens arrays can be used in light field display devices to extend light field data to the angular domain, thereby recreating an approximation of the original light field. These types of light field displays use microlens light field images corresponding to the geometry of the display lens array. The microlens light field format interleaves samples of individual views under each microlens. For example, Ostendo has developed a light field display supporting 50×50 = 2500 samples under each microlens. See, for example, the following article: Zahir Alipasian & Hussein El-Ghoroury, Small Form Factor Full Parallax Tiled Light Field Display, Proceedings of SPIE (March 17, 2015). The term "full-light" is also used to describe this format, rather than the term "microlens."

[0116] In some cases, delivering data for such a large number of views without significant data optimization may be impractical. Small lens light field data can be optimized by converting the contents of the small lens light field to a multi-view light field before compression (see Ebrahimi), and then converting it back to the small lens format at the receiver if necessary. This optimization was developed for multi-view video and image array light fields. However, the processing power and memory bandwidth required for this conversion may make such optimization impractical.

[0117] Several articles describe specialized optimizations developed for microlens light fields. Given the similarity between microlens images and frame-to-frame spatiotemporal compression, as understood in Viola, Irene et al., *Objective and Subjective Evaluation of Light Field Image Compression Algorithms*, 32nd Picture Coding Symposium (2016), many of these articles introduce separate compression steps. However, many of these articles are not considered to account for redundancy in the light field data if the display device uses only one or a few viewpoints, or if view fidelity can be adjusted based on the fovea. For some embodiments, the fidelity of visual elements can be adjusted if the visual object is within the viewer's fovea and / or peripheral vision.

[0118] Depending on the application, many methods for compressing light field data may not be optimal, especially when, for example, the client viewpoint can be used to guide the rendering and optimization process. To optimize rendering and data distribution, depending on the viewing client's ability to estimate the data to be used, only a minimal subset of the full light field data can be generated and transmitted. For small-lens light fields, the light field data can be split into separate streams in a way that achieves fidelity to non-uniform light field across the light field region.

[0119] Light fields generate a large amount of data, and optimizing data generation, storage, and transmission can be helpful. The microlens format of a light field comprises a large number of local views sampled at high angular resolution. These local views are collected as an array, the format of which can be determined by a microlens array used to capture the data or by the microlens optics used by the light field display. Each individual local view in a microlens light field corresponds to a single lens in the lens array. Because the microlens light field format differs from image array-type light fields, specialized methods can be used to optimize rendering and transmission.

[0120] In some embodiments, the transmission of microlens light field data is optimized by dividing the microlens light field data into several substreams. Dividing it into substreams enables adaptive streaming systems that reduce the amount of light field data sent for content delivery.

[0121] In some embodiments, the content server can divide the microlens light field data into several substreams, thereby optimizing the amount of light field data used by the client to synthesize a view or display the light field on a light field display with specific display capabilities. On the server side, the microlens light field data can be optimized by estimating the number of full microlens views used to reproduce the full light field in high quality. The optimized microlens data can be divided into several substreams, allowing the viewing client to selectively control the light field quality across the entire light field area, while also adapting the amount of data transmitted to the available transmission bandwidth and client-side processing resources.

[0122] In the primary embodiment, the viewing client is a display device having a lens array that produces images with high angular resolution. This type of display can be a large fixed light field display or a single-user light field HMD, such as those disclosed in the following article: Lanman, Douglas and Luebke, David, Near-Eye Light Field Displays, 32.6 ACM Transactions on Graphics (TOG) 220 (2013). Both types of devices can utilize user and / or eye tracking to adjust display fidelity to match the viewer's visual perception characteristics.

[0123] Some embodiments utilize a client-pull model similar to MPEG-DASH, where the server provides the client with a manifest file indicating available data streams. The client performs performance analysis to determine bandwidth and processing limitations and adjusts data transmission accordingly. In adapting data transmission, the client can prioritize streams to be pulled based on user tracking and content analysis, maximizing perceived quality of experience (QoE).

[0124] Transmitting a full-minimum lens optical field containing full-angle information for each miniature lens can utilize a wide transmission bandwidth. Much of this data may be redundant and can be reproduced by the receiving client from more sparsely sampled data.

[0125] Furthermore, if the quality is dynamically adjusted based on content characteristics and the viewer's perceptual properties, the viewing client can adjust the quality of the light field rendering to be non-uniform across the entire light field without reducing the QoE.

[0126] Some implementations allow clients to retrieve only a portion of the microlens light field data for use at a specific time, thereby reducing the amount of data transmitted. Furthermore, because the content server divides the microlens light field data into multiple substreams, the data has been optimized by reducing the number of microlens views, where full-angle data is used to reproduce the full light field.

[0127] Figure 5 This is a system diagram illustrating a set of example interfaces for a viewing client according to some embodiments. In some embodiments, the viewing client 510 may interface with a display 516 and one or more sensors. The viewing client 510 may include a local cache memory 514. In some embodiments, one or more displays 516 and one or more sensors may be located locally. In other embodiments, one or more displays 516 and one or more sensors (such as tracking and input sensors 518) may be located externally. In some embodiments, the viewing client 510 may locally store content in the local cache 514. In some embodiments, the viewing client 510 may perform a streaming adaptation and rendering process 512. The viewing client 510 may connect to a content server 502 via a network 508 (e.g., a cloud network). In some embodiments, a subsampled set of Media Presentation Description (MPD) files and spatial data (e.g., various resolution and bitrate versions of spatial data) 506 may be stored on the content server 502. In some embodiments, the content server 502 may store raw microlens light field content 504 in a database.

[0128] In some embodiments, content server 502 may generate multiple subsample sets from the raw light field content of microlens format 504. In addition to the subsample sets, content server 502 may generate metadata describing the properties of the raw light field data and the available subsample sets. This metadata may be stored in a description file known as a Media Presentation Description (MPD) for the MPEG-DASH protocol.

[0129] Figure 6 This is a message sequence diagram illustrating an example process of a typical session sequence according to some embodiments. Figure 6The diagram illustrates a communication sequence between viewer client 602 and content server 604 in a typical usage session. In some embodiments, content server 604 may render a 606 small lens light field subsample set and mapping metadata, and may generate a 610 MPD. Viewer client 602 may send a 608 content request to content server 604, and content server 604 may send back a 610 MPD file. In some embodiments, content server 604 may divide the light field content into one or more light field regions. Viewer client 602 may select a 612 subsample set for each light field region. Viewer client 602 may request a 614 subsample set from content server 604, and content server 604 may send a 616 subsample set and mapping metadata of a video sequence repackaged into a smaller, denser light field. Viewer client 602 may update the 618 user viewpoint and reconstruct the full small lens light field from the received data, taking into account the temporal variability of the light field data. Viewer client may build a 620 motion model of the viewpoint motion and predict the viewpoint position for the next time step. The viewing client can analyze 622 content based on predicted view locations and performance metrics, and can adjust subsample set requests.

[0130] In some embodiments, the content server may send a media manifest file to the viewing client containing multiple subsampled microlens representations describing multiple portions of the light field video content. In some embodiments, the content server may receive information from the viewing client indicating a subsampled microlens representation selected from the multiple subsampled microlens representations. In some embodiments, the content server may send the selected subsampled microlens representation to the viewing client.

[0131] Figures 7A-7B This is a flowchart illustrating an example process of a content server procedure according to some embodiments. In a microlens light field format, views with angular variations are interleaved for each microlens. The interleaving characteristic of angular resolution makes it difficult to adapt the microlens format in terms of angular resolution. However, due to the direct relationship between microlens density and spatial resolution, the spatial resolution can be adjusted. In some embodiments, the content server divides the entire microlens light field into smaller sub-regions and subsamples the microlens data of each sub-region at multiple sampling rates to produce multiple subsampled subsets, which the requesting client can select to adapt the data stream to each client's constraints and requirements.

[0132] This paper further describes example preprocessing and rendering of small-lens light field video for adaptive streaming. Additionally, it describes example structures and data included in the MPD for transmitting light field specifications and available subsets from the server to connected clients. Aside from light field data processing and specialized metadata formats, the content server can operate, for example, similar to a video content server that uses adaptive content distribution of MPEG-DASH type to deliver data to clients. Figures 7A-7B An example of processing performed by the content server is shown.

[0133] The server process can perform content preprocessing, wherein the server generates multiple subsets of light field data and metadata describing those subsets. The first step of this process is to select a full small lens light field specification (702). The server can generate light field data to be streamed, for example, from: an existing small lens light field (708), an image array light field, or spatial data (704) in a full 3D format, such as a real-time 3D scene or point cloud. The server can determine (706) whether to use the original input small lens light field. For image array light fields and full 3D scenes, the server renders and / or transforms the data to a small lens format (710). To render and / or transform the data (710), the server uses a specific small lens light field specification, which can be selected (712) and / or set at a first time step and can be saved to an MPD. For existing small lens light fields, the specification can be collected and saved to an MPD. The region division of the full small lens light field (712) can be determined. Samples to be used within the region (714) can be determined, and a set of subsamples with the selected samples (716) can be generated. This article describes this example process of generating subsample sets in more detail.

[0134] The resulting subsample set 718 can be compressed and stored 720 in preparation for streaming. The subsample set and metadata 722 of the video file can be compressed using appropriate existing encoders for streaming video formats, and various versions can be generated using varying compression and resolution. Information about the resulting subsample set versions is collected in the MPD.

[0135] If the server has already produced a set of subsamples that can be streamed to the viewing client, and if the server has compiled (or collected 724) information about the microlens light field specification and the available set of subsamples into MPD 726, then the server can be prepared to stream content to the client in response to a client request. The server can switch to runtime mode (primarily...) Figure 7BThe process waits for a client request at 752. If the client connects and requests content, the MPD is transmitted to the client at 754. The client can select which parts of the content to request streaming based on the MPD. In some embodiments, streaming 756 may continue until a session end request is made at 758. In some embodiments, the runtime process continues until a processing end request is made at 760, at which point the process may end at 762.

[0136] In some embodiments, the content server process may include: subsampling the entire microlens dataset to obtain multiple sparsely sampled regions; generating a media presentation description (MPD) specifying the original microlens data attributes and the available set of subsamples; waiting for a content request; sending the MPD to a client requesting the content; and streaming the requested set of subsamples to the client.

[0137] In some embodiments, a sub-sampling rate can be selected for the microlens representation. In some embodiments, the optical field spatial resolution can be selected, and the optical field content can be divided into portions corresponding to the selected optical field spatial resolution.

[0138] Figure 8 This is a schematic diagram illustrating an exemplary full microlens light field divided into multiple regions according to some embodiments. In some embodiments, the light field data is preprocessed before the content server distributes the light field data in an adaptive microlens format. During preprocessing, the microlens light field can be divided into multiple regions, and a subsample set can be generated for each region. For each region, multiple versions of the available light field data can exist, allowing clients to request light field data for different portions of the light field with non-uniform fidelity.

[0139] Figure 8 A full microlens light field frame divided into several regions is shown. For example, the regions can be divided such that there is a predetermined number of individual microlens subviews within a single region, or the regions can be divided to have a predetermined number of regions. In some embodiments, the division can be accomplished using non-uniform region sizes, or region sizes or areas can be dynamically assigned, and can be assigned based on, for example, content analysis. Figure 8 In the example shown, the full microlens light field frame is divided into a 7×7 region array 802, and in each region, there is a 6×4 microlens subview array 804.

[0140] Figures 9A-9C This is a schematic diagram illustrating an exemplary set of subsamples with varying subsample densities at a first time step, according to some embodiments. Figures 10A-10C This is a schematic diagram illustrating an example subsample set with varying subsample densities at a second time step, according to some embodiments.

[0141] If the content has been divided into multiple regions, the content server can generate a subsample set for each region. The content server can generate several versions of the light field content, where the subsample density of the light field data differs for each region. The subsample set can be generated by reducing the number of individual small lens subviews included with the data, thereby omitting one or more of these subviews from the full light field data. For a given subsample density, the number of small lens subviews to be stored is fixed. The actual positions of the small lenses can change (or have temporal variations) throughout the small lens light field video or can be fixed. The selection of small lens positions included in the subsample set can be based on subview analysis, which estimates which subviews can be used for the most accurate interpolation of the omitted subviews. This analysis can be performed frame-by-frame, which can cause dynamic changes in the positions of the subviews used in the subsample set.

[0142] Figures 9A-9C Figures 10A-10C show the small lens sub-viewpoint locations selected to be included in the sub-sample set, which has three different sampling densities and locations at two different time steps. Figure 9A and 10A Examples of subsample sets 902 and 904 at the first and second time steps are shown, with subviews 904 and 1004 having a subsample density of 1 / 6. Figure 9B and 10B Examples of subsample sets 932 and 1032 at the first and second time steps are shown, with subviews 934 and 1034 having a subsample density of 3 / 8. Figure 9C and 10C Examples of subsample sets at the first and second time steps are shown, with subviews 964 and 1064 having a subsample density of 1 / 2.

[0143] Figure 11A-11C This is a schematic diagram illustrating an example subview of a set of subsamples selected for packaging into a dense integral image according to some embodiments. Figure 11A An example of subsample set 1102 is shown, which has a subview 1104 that is packed into a dense integral image with a subsample density of 1 / 6. Figure 11B An example of subsample set 1132 is shown, which has a subview 1134 that is packed into a dense integral image with a subsample density of 3 / 8. Figure 11C An example of subsample set 1162 is shown, which has a subview 1164 that is packed into a dense integral image with a subsample density of 1 / 2.

[0144] The subviews selected for the subsample set can be rearranged into a dense array to form a dense integral image, but with a reduced number of subviews. For each subsample set, the individual frames formed by the packed subviews can be compiled into the video file along with a metadata file indicating the mapping from the microlens positions of the packed integral image to the original microlens subview positions in the original microlens light field integral image. The mapping metadata can be compiled into an MPD as part of each representation block to indicate the mapping of the subsample set. Furthermore, the metadata header can indicate the microlens size for a specific resolution of the available subsample set. Listing 1 illustrates a packed 2×2 subsample array according to some embodiments (…). Figure 11A Mapping to the original image ( Figure 9A and 10A Example metadata for ). For this example, the location is given using an xy coordinate system (x, y) with the origin at the top right corner.

[0145] Resolution 1

[0146] Based on the sample size of each pixel: n1 x m1 pixels

[0147] Resolution 2

[0148] Based on the sample size of each pixel: n² x m² pixels

[0149] Time step 1

[0150] Packaging location 1,1 Original location 1,1

[0151] Packaging location 2,1 Original location 5,2

[0152] Packaging location 1, 2; Original location 2, 3

[0153] Packaging location 2,2 Original location 6,4

[0154] Time step 2

[0155] Packaging location 1,1 Original location 4,1

[0156] Packaging location 2,1 Original location 3,2

[0157] Packaging location 1,2 Original location 5,2

[0158] Packaging location 2,2 Original location 5,3

[0159] List 1. Mapping Metadata

[0160] Some embodiments may use subsampling modes to simplify the description of sample locations. For example, mapping metadata may indicate the time step and subsampling mode, rather than indicating the mapping of individual microlens samples from the packaged format to the original complete microlens array. The regular subsampling mode and the individual sample pixel size configuration can be identified in the header of the mapping metadata.

[0161] Other video formats and codecs can be used to encode and compress video files compiled from subsample sets. In some embodiments, several versions of subsample set video files with different resolutions can be generated. Reducing the resolution of the small lens integral image effectively reduces the angular and spatial resolution of the light field that can be reconstructed from the data.

[0162] Figure 12 This is a data structure diagram illustrating an example MPEG-DASH Media Presentation Description (MPD) according to some embodiments. The client-side pull model can use... Figure 12 The general structure of the MPEG-DASH Media Presentation Description (MPD) shown is because the file format used to send the overall media description can be downloaded by the viewing client to initialize the streaming session.

[0163] Figure 12 This is a data structure diagram illustrating an example MPEG-DASH Media Presentation Description (MPD) according to some embodiments. Figure 12 The structure of an MPEG-DASH Media Presentation Description (MPD) file is shown. This file format can be used to send an MPD from a content server to a viewing client. In some embodiments, an MPD file can be sent to initiate a streaming session. The MPD file 1202 may include one or more periods 1204, 1226 as top-level entities. Periods 1204, 1226 may include the start time and duration of the content. Each period provides information about a single small lens light field scene, such as the light field using a fixed camera position and / or light field specification. The entire user experience may include several scenes, each corresponding to a period block.

[0164] The time periods 1204 and 1226 may include one or more adapter sets 1206 and 1224. The first adapter set 1206 may list each available small lens light field subsample set for each region of the light field. After the first adapter set 1206, which describes the overall structure of the subsample sets created for each region, the second and subsequent adapter sets 1224 may indicate details about the subsample sets for each region.

[0165] Many adapter sets 1206, 1224 can contain media streams. Adapter sets 1206, 1224 can contain one or more representations 1208, 1222. Representations 1208, 1222 can include one or more encodings of the content, such as 720p and 1080p encoding. Representations 1208, 1222 can include one or more segments 1210, 1220. Segments 1210, 1220 are media content data that a media player (or viewing client) can use to display the content. Segments 1210, 1220 can include one or more sub-segments 1216, 1218, which are represented by representation fields to indicate sub-representations 1212, 1214. Sub-representations can contain information applicable to a specific media stream.

[0166] Figure 13 This is a data structure diagram illustrating an example media presentation description (MPD) with example microlens light field description (one or more), subsample set, resolution, and bit rate according to some embodiments. Figure 13 This demonstrates how to organize MPD 1302 data within the general MPD structure of MPEG-DASH.

[0167] The next level after time periods 1304 and 1346 is the microlens light field specification 1306. Light field specification 1306 can indicate: the division of the full microlens light field into multiple regions, the individual microlens images in each region, the spatial and angular resolution of the microlenses, the location and measurement of the microlens light field capture and / or rendering settings, and an overview of the scene layout, the size and placement of scene elements.

[0168] Each subsample set 1308, 1340, 1342 may have metadata 1338, which describes the mapping between the densely packed subsampled microlens image positions in the video file and the microlens positions in the original full microlens light field. Within each subsample set 1308, 1340, 1342, multiple versions of the same data encoded in different ways may be provided. Each version may have different resolutions 1310, 1312, and different resolution versions 1310, 1312 may provide the same resolution content using compression with different bit rates 1314, 1316, 1318, or different supported codecs.

[0169] Figure 13Each encoded version, referred to as bitrate 1314, 1316, and 1318, provides a link to the video file, where subsampled microlens images can be repackaged into dense microlens light fields. Each video file can be temporally divided into segments, allowing clients to switch between different versions within a single time period. Sub-segment blocks of the MPD provide URL links to the actual video data 1326, 1328, 1330, 1332, 1334, and 1336. These URLs can be associated with time steps 1320, 1322, and 1324. Some time periods 1304 and 1346 may include audio data 1344.

[0170] Will Figure 12 and 13 Together, a subsample set can correspond to an MPEG-DASH adaptation set, and the resolution under a given subsample set can correspond to an MPEG-DASH representation and a segment. In some embodiments, a media block can correspond to an MPEG-DASH representation, and the time step can correspond to a subrepresentation.

[0171] Figure 14 This is a process diagram illustrating an example small lens array reconstruction process according to some embodiments. Figure 14 An example process executed by a client, such as a viewing client, is illustrated. In some embodiments, a user launches an application implementing the viewing client, and the process is executed by the viewing client. If the user launches the application, the user can specify the content to be viewed. In some embodiments, the content may be delivered as a link to content residing on a content server. The link to the content may be a URL identifying the content server and a specific MPD file. The viewing client application may be launched, for example, by an explicit command from the user, or automatically by the operating system based on a request identifying the content type and an application associated with that specific content type. In some embodiments, the application is a standalone application. In some embodiments, the viewing client may be integrated with a web browser or social media client. In some embodiments, the application may be part of the operating system.

[0172] If the viewing client application is launched, the application can initialize sensors (e.g., geolocation and imaging sensors) used to track the device, the user, and / or the user's gaze direction. Based on display specifications, tracking settings, and application-specific settings, the viewing client can determine the initial set of subsamples to request from the server. In some embodiments, devices such as HMDs, the user (e.g., for use with a standalone display), and / or the user's eyes can be used to determine the user's gaze direction. In some embodiments, gaze direction can be used to determine which content areas are seen by the user's fovea and which content areas are seen in the user's peripheral vision. In some embodiments, such determination can be used to control fidelity levels and streaming adaptation.

[0173] If the selected subsample set has been requested, the client can download the associated video and metadata, and sequentially download sub-segments from the server. Once the first sub-segment has been received, the client can begin runtime operations. During runtime, the client can update the viewpoint based on the tracking and user input (e.g., the user adjusting the viewpoint using user interface controls). Using the updated viewpoint, the client can render the light field to a display-specific format based on the received data.

[0174] In some embodiments, Figure 14 An example rendering process for an example 2×2 array subsample set 1410 is illustrated. A client may receive the subsample set 1410 and set-specific mapping metadata 1412 as described at 1402. In some embodiments, the client can use the subsample set-specific mapping metadata 1412 to map the small lens image from received subsampled segments to the small lens positions of the full small lens light field 1414 as described at 1404, 1406. For some embodiments, the missing small lens image 1416 from the received subsample set can be reconstructed by interpolating 1408 between the received small lens images to produce, for example, the full small lens light field of small lens image 1418, such as... Figure 14 The example process is illustrated below. In some embodiments, the full microlens 1420 generated in rendering may have specifications different from, for example, the original microlens light field initially used by the server to create the subsample set. For example, a viewing client may generate a microlens light field corresponding to the display device specifications during the rendering process.

[0175] In some embodiments, the client may receive a packaged subsample set and mapping metadata. Individual microlens samples can be transformed from their packaged positions to their original microlens positions in the original microlens array based on the correct time-step mapping indicated in the metadata. In some embodiments, microlens samples omitted from the subsample set can be reconstructed by interpolation from the transmitted samples. In some embodiments, a full microlens light frame can be generated by repeating this process for each light field region in the frame. In some embodiments, the reconstructed full microlens light frame can be displayed by a viewing client.

[0176] In some embodiments, interpolating a view from a retrieved subsampled microlens representation using the description of the microlens representation in the manifest file may include: unpacking the retrieved subsampled microlens representation to the original microlens positions of the portion of the light field video content indicated in the manifest file; and interpolating microlens samples omitted from the retrieved subsampled microlens representation. In some embodiments, generating a complete light field region image of a portion of the light field video content from the interpolated view of the retrieved subsampled microlens representation.

[0177] Figure 15 This is a flowchart illustrating an example process for a viewing client according to some embodiments. During the runtime process, the client can adapt to the selection of subsample sets. According to this example, the client can decide on a region-by-region basis which and how many subsample sets to pull, and at what resolution and compression quality to pull the light field. In some embodiments, different compression qualities and different compression techniques may be used for each region. In some embodiments, the selection may be made based on viewpoint motion, content complexity, display capabilities, current network and processing performance, or any other local criteria.

[0178] In some embodiments, the example viewing client process may include requesting content from a content server (1502). The viewing client may receive MPD (1504) in the response. The viewing client may initialize tracking (1506), which may include, for example, eye tracking, gaze tracking, user tracking, and / or device tracking). In some embodiments, an initial viewpoint may be set. The viewing client may select an initial set of subsamples to request (1508) and request these selected subsamples from the content server (1510). The requested subsamples and mapping metadata may be received by the viewing client from the content server (1512). The viewpoint may be updated (1514). The viewing client may unpack the received microlens subsamples from the packet transmission format into a full microlens image, which can be done by using mapping metadata indicating the original position in the full microlens image (or, in some embodiments, a frame or region). Missing microlens samples may be interpolated, and fidelity may be adapted based on the current viewpoint. In some embodiments, the resolution of the viewing area may be adjusted if missing microlens samples are interpolated. In some embodiments, if the region is not located in the fovea region of the retina, the original full-lens light field of the region can be reconstructed at a lower spatial resolution. The full-lens light field image can be rendered and displayed 1516. The subsampling set selection 1518 can be updated based on tracking data (which may include eye tracking, gaze tracking, user tracking, and / or device tracking), user input, content analysis, and / or performance metrics. In some embodiments, regions farther from the fovea can be set to have a lower subsampling rate based on the gaze direction detected by the tracking. In some embodiments, the subsampling rate for all regions can be adjusted to lower or higher based on user input or performance metrics. If the end of processing is not received 1520, the viewing client process can be repeated by requesting an updated selection of the subsampling set. Otherwise, the process ends 1522.

[0179] In some embodiments, the viewing client can update the user's viewpoint and adjust the selected microlens representation for the updated viewpoint. In some embodiments, the viewing client can predict changes in the user's viewpoint and adjust the selected microlens representation for the predicted viewpoint. In some embodiments, the viewing client can select a subsampling rate for the selected microlens representation such that the subsampling rate uses the predicted viewpoint. For example, if the user's viewpoint changes to a point to the left of the current viewpoint, regions of the light field image closer to the new viewpoint can have an increased associated subsampling rate to generate a higher quality image in the region surrounding the new viewpoint. Similarly, regions farther from the new viewpoint can have a decreased associated subsampling rate. In some embodiments, the spatial resolution of the light field can be adjusted to improve performance metrics of the interpolated view. For example, the spatial resolution can be increased for regions and portions of the region closer to the user's viewpoint, and decreased for regions and portions of the region farther from the current viewpoint. These changes can result in, for example, higher user satisfaction, higher image resolution, and / or higher microlens density in the region surrounding the current user's viewpoint.

[0180] Figure 16A This is a schematic diagram illustrating an example multi-view array representation according to some embodiments. Figure 16B This is a schematic diagram illustrating an exemplary representation of a full microlens according to some embodiments. The Ebrahimi test dataset m41995 is used for... Figures 16A to 18B . Figure 16A and 16B The diagram shows two equivalent representations of the sample content: multiple views (multi-view array representation 1602) and a small lens (full small lens representation 1652). For Figure 16B Multiple views are staggered pixel by pixel.

[0181] Figure 17A This is a schematic diagram illustrating an example multi-view array representation of the illustrated ROI according to some embodiments. Figure 17B This is a schematic diagram illustrating an exemplary microlens representation of the illustrated ROI according to some embodiments. Regions of interest (ROIs) 1704 and 1754 are shown as small dashed rectangles at the center of each representing 1702 and 1752. Figure 17A and 17B Two equivalent representations of the sample content are shown: multiple views and a small lens. If the client wants a finite region of interest (ROI), this finite ROI can be indicated in the MPD.

[0182] Figure 18A This is a schematic diagram illustrating an example multi-view array representation with the illustrated ROI and selected views according to some embodiments. Figure 18BThis is a schematic diagram illustrating an exemplary total microlens representation having the illustrated ROI according to some embodiments. Figure 18A and 18B Two equivalent representations of the sample content are shown: multi-view 1802 and small lens 1852. (For...) Figure 18A The multi-view array shows a limited selection of views. Figure 18A In the diagram, the selected view is shown as a dashed ROI rectangle 1804, and the unselected view is shown as a solid ROI rectangle 1806. Figure 18B The selected view with dashed ROI rectangle 1854 is shown. If the client wants a limited number of views (e.g., only dashed ROI rectangles) corresponding to the subsamples represented by the small lens, that limited number of views can be indicated in the MPD.

[0183] Figure 19 This is a process diagram illustrating an example subsampling process for selecting a small lens view according to some embodiments. Figures 18A-18B The diagram shows indications regarding subsets of views. Small lens representations are not shown to avoid confusion regarding interlacing. Figure 19 The corresponding ROI clipping and subsampling are shown in the figure.

[0184] exist Figure 19 On the left, a full image 1902 with 425×425 samples is shown. A region of interest (ROI) 1904 is shown at the center of the full image 1902. ROI 1904 is extracted from the full image 1902. In this example, the top-left corner of ROI 1904 is at coordinates (200, 200), and the bottom-right corner is at coordinates (325, 325). The cropped image 1906 has 125×225 samples. The cropped image 1906 is divided into 5 vertical regions and 5 horizontal regions (a total of 25 regions). The cropped image 1906 is subsampled to constrain the view, and the subsample set 1908 is 25×133 samples with 1 vertical region and 3 horizontal regions (a total of 3 regions).

[0185] According to some embodiments, sample MPDs are shown in Listing 2, giving different ROI and microlens sampling options. This list shows three different LF microlens adapter sets: Full, Center, and Left. The range of microlens views is expressed for each different set. Note that for a given total resolution, fewer views give more pixels per view. Note that subranges can overlap, thus giving different representations. In addition to range, microlens density is also specified. Only the horizontal direction is specified by a [N, 1] density. Conventional DASH rate and resolution adapters can be used within each view category. An example of UHD / HD is shown in the "Center" adapter set. The MPD examples shown in Listing 2 correspond to... Figure 19 The subsampling process is shown in the figure.

[0186]

[0187] List 2. Example MPD List

[0188] MPD can include details represented by the entire light field via microlenses (such as the NxM view position). For

[0189] A subset of elements within a finite horizontal and vertical range can be indicated in the MPD. The fit set within the MPD can include multiple subsets of microlens representations, which have varying microlens representation angular positions and ranges.

[0190] In some embodiments, within the context of adaptive streaming, the content server can prepare a reduced version of the content relative to the entire scene. This reduction may include a limited number of image ROIs and a limited number of viewing directions. The viewing client can select this based on multiple factors, including, for example, display capabilities, viewer gaze, and bandwidth.

[0191] The number and detail of the microlens subviews can depend on the desired characteristics of the display. Displays implementing simple multi-view representations can choose a few sparse microlens views. Displays capable of providing smooth motion parallax through native multi-view or viewer tracking can choose a medium density of microlens views. Displays providing natural focus cues can use a high microlens density. For another example, if the display only provides views that differ in the horizontal direction, such as in… lookingglassfactory <dot> com< / dot> If a viewing glass display is available, the option to display only horizontal parallax can be selected.

[0192] In some embodiments, the ROI can be selected based on the current or predicted viewer gaze. This selected ROI can be used in conjunction with an explicit ROI representation for an individual stream, or it can be enabled via chunk-based streaming of microlens images. Typically, high-density microlens data can be selected centered on the viewer's gaze location, while lower-density microlens or planar 2D data can be selected away from the viewer's gaze location.

[0193] Adaptive streaming systems can use bandwidth measurements to select between different resolutions or bitrate representations of content. For streaming microlens content, in addition to the aforementioned display capabilities, bandwidth limitations may lead clients to choose lower microlens densities, allowing them to rely on view interpolation to generate untransmitted intermediate views. Therefore, microlens density is an additional factor in adapting to spatial resolution.

[0194] In some embodiments, the viewing client can use its capabilities to select a microlens representation. For example, the viewing client can select a microlens representation with a higher resolution image because the viewing client's display is capable of handling higher resolution images. Other examples include bit rate, available bandwidth, bandwidth to be consumed, and the maximum microlens density supported by the viewing client's display. In some embodiments, selecting a subsampled microlens representation may include selecting a subsampled microlens representation with a density higher than a threshold supported by the client.

[0195] Figure 20 This is an image illustration showing an example full microlens image with 425×425 samples and 5×5 views according to some embodiments. Figure 20 The example in the example is a full microlens image 2002. As mentioned above regarding the use of viewer gaze for a specific microlens stream, chunk-based streaming can be used in some embodiments to generate multiple content representations with different microlens densities. Each representation is the entire image of the scene, but is broken down into tiles for distribution. Clients can select tiles around the viewer's gaze point to be represented with a high microlens density, and select tiles outside the viewer's gaze area to be represented with a lower microlens density, such as a planar 2D representation. In some embodiments, the stream may not use ROI declarations and rely on chunking to resolve a limited ROI, but for streams with different microlens densities, ROIs can be provided. Figure 20-23 The image shows a fully subsampled microlens image that can be used as a source for adaptive streaming. Figure 20 An example full-microlens image with 425×425 samples and 5×5 views is shown.

[0196] Figure 21 This is an image illustration showing an example subsampled microlens image with 255×255 samples and 3×3 views according to some embodiments. Figure 21 yes Figure 20 Example subsampling version 2102.

[0197] Figure 22 This is an image illustration showing an example subsampled microlens image with 255×85 samples and 1×3 views according to some embodiments. Figure 22 yes Figure 20 Example subsampling version 2202.

[0198] Figure 23 This is an image illustration showing an example subsampled microlens image with 85×85 samples and 1×1 views according to some embodiments. Figure 23 yes Figure 20 Example subsampling version 2302.

[0199] According to some embodiments, the example MPD in List 3 is described based on microlens density. Figures 21 to 23 The diagram shows various representations of different microlens densities. Within a density, both resolution and bit rate representations can exist. Clients can select tiles for different microlens density representations based on the criteria previously outlined.

[0200]

[0201] List 3. Example MPDs with different microlens density representations

[0202] Figure 24A -C is a schematic diagram illustrating an example small lens light field subsampled at two subsampling densities according to some embodiments. Figure 24A The image shows a full microlens light field image of a 7×7 array divided into regions 2402. For example, a 6×4 array 2404 of region 10: microlens subview is shown. Figure 24B The first subsample set 2432 with a subsample density of 1 / 6 is shown. Figure 24C The second subsample set 2462 with a subsample density of 3 / 8 is shown.

[0203] for Figures 24A-24C The exemplary small lens light field shown can be described in some embodiments as a pseudo-MPD with the structure shown in Listing 4. This pseudo-MPD describes the light field structure in adapter set 1. Adapter sets 22 and 23 describe two different sampling sets for region 10. Figure 24B This corresponds to the exemplary time step 1 under adaptation set 22. Figure 24C This corresponds to the example time step of 1 under adaptation set 23. For both examples, the (x, y) coordinates use the origin at the top left corner. The example codecs listed below have different names for the codecs, although codecs with different names can use the same codec (e.g., "Codec 5" and "Codec 4" shown below can use the same codec).

[0204] Time period

[0205] Adaptation Set 1: Light Field Scene Description:

[0206] Full microlens light field resolution, region definition, region specifications: original resolution, number and location of microlens images, angular resolution of microlens images. A list of regions and available subsample sets for each region. ...

[0208] Adaptation set 22: Region 10, Subsampling set 1 (2 x 2 small lens sampling)

[0209] Mapping metadata: A file containing a time-stamped coordinate transformation that maps the miniature lens image from the packed miniature lens light field of the subsampled set to the original miniature lens image position in the full miniature lens light field. Pseudo-example of data:

[0210] Resolution 1

[0211] Based on the sample size of each pixel: 16 x 16 pixels

[0212] Resolution 2

[0213] Based on the sample size of each pixel: 12 x 12 pixels

[0214] Time step 1

[0215] Packaging location 1,1 Original location 6,1

[0216] Packaging location 2,1 Original location 2,2

[0217] Packaging location 1, 2; Original location 5, 3

[0218] Packaging location 2,2 Original location 1,4

[0219] Time step 2

[0220] Packaging location 1,1 Original location 4,1

[0221] Packaging location 2,1 Original location 3,2

[0222] Packaging location 1,2 Original location 5,2

[0223] Packaging location 2,2 Original location 5,3 ...

[0225] Resolution 1: 1024 x 768px

[0226] Bit rate 1: Encoding using codec 5 requires a transmission capacity of 5.

[0227] Mbps

[0228] Bit rate 2: Encoding using codec 4 requires a transmission capacity of 4.

[0229] Mbps

[0230] Resolution 2: 800 x 600px

[0231] Bit rate 1: Encoding using codec 3 requires a transmission capacity of 3.

[0232] Mbps

[0233] Bit rate 2: Encoding using codec 2 requires a transmission capacity of 2.

[0234] Mbps

[0235] Adaptation set 23: Region 10, Subsampling set 2 (3 x 3 small lens sampling)

[0236] Mapping metadata: A file containing a time-stamped coordinate transformation that maps the miniature lens image from the packed miniature lens light field of the subsampled set to the original miniature lens image position in the full miniature lens light field. Pseudo-example of data:

[0237] Resolution 1

[0238] Based on the sample size of each pixel: 15 x 15 pixels

[0239] Resolution 2

[0240] Based on the sample size of each pixel: 11 x 11 pixels

[0241] Time step 1

[0242] Packaging location 1,1 Original location 1,1

[0243] Packaging location 2,1 Original location 3,1

[0244] Packaging location 3,1 Original location 5,1

[0245] Packaging location 1,2 Original location 2,2

[0246] Packaging location 2,2 Original location 6,2

[0247] Packaging location 3,2 Original location 3,3

[0248] Packaging location 1,3 Original location 5,3

[0249] Packaging location 2,3 Original location 1,4

[0250] Packaging location 3,3 Original location 6,4

[0251] Time step 2

[0252] Packaging location 1,1 Original location 2,1

[0253] Packaging location 2,1 Original location 4,1

[0254] Packaging location 3,1 Original location 5,1

[0255] Packaging location 1,2 Original location 4,2

[0256] Packaging location 2,2 Original location 6,2

[0257] Packaging location 3,2 Original location 1,3

[0258] Packaging location 1,3 Original location 4,3

[0259] Packaging location 2,3 Original location 6,3

[0260] Packaging location 3,3 Original location 5,4 ...

[0262] Resolution 1: 1920 x 1080px

[0263] Bit rate 1: Encoding using codec 9 requires a transmission capacity of 9.

[0264] Mbps

[0265] Bit rate 2: Encoding using codec 8 requires a transmission capacity of 8.

[0266] Mbps

[0267] Resolution 2: 1280 x 720px per subview

[0268] Bit rate 1: Encoding using codec 6 requires a transmission capacity of 6.

[0269] Mbps

[0270] Bit rate 2: Encoding using codec 5 requires a transmission capacity of 5.

[0271] Mbps ...

[0273] List 4. Example MPD

[0274] In some embodiments, the description of the microlens representation (which may be part of the MPD or manifest file) may include information about at least one of the range and density of the microlens representation. In some embodiments, interpolation views from the subsampled microlens representation may use the description of the microlens representation in the manifest file. In some embodiments, selecting a subsampled microlens representation includes selecting the subsampled microlens representation with the largest range. In some embodiments, selecting a subsampled microlens representation includes selecting the subsampled microlens representation with the highest density. In some embodiments, selecting a subsampled microlens representation is based on at least one of the following: the density of the selected subsampled microlens representation, or the range of the selected subsampled microlens representation.

[0275] In some embodiments, a process may include selecting a light field spatial resolution; dividing the light field content into a plurality of portions corresponding to the light field spatial resolution; and selecting a microlens image for at least one frame of at least one subsampled microlens representation of at least a portion of the light field content, such that selecting a subsampled microlens representation for at least a portion of the light field content selects a corresponding subsampled microlens representation, and such that the corresponding microlens image is used in an interpolated view from the subsampled microlens representation.

[0276] Figure 25 This is a message sequence diagram illustrating an example process for adaptive light field streaming using estimated bandwidth and view interpolation, according to some embodiments. In some embodiments, the example process may include content server 2502 generating 2506 a subsampled microlens representation and a descriptive MPD. In some embodiments, the example process may also include viewing client 2504 tracking 2508 viewer gaze (or user position, device position, or eye position). In some embodiments, the example process may also include client 2504 sending 2510 a content request to content server 2502. In some embodiments, the example process may also include content server 2502 responding 2512 using the MPD. In some embodiments, the example process may also include client 2504 estimating 2514 the bandwidth available and / or predicting the bandwidth that will be used by the requested content. In some embodiments, estimating the bandwidth may include determining an estimated bandwidth available for streaming light field video content. In some embodiments, the example process may also include viewing client 2504 selecting 2516 a microlens representation. In some embodiments, the example process may also include viewing client sending 2518 a light field representation request to content server 2502. In some embodiments, the example process may further include content server 2502 retrieving 2520 and transmitting 2522 of the requested representation fragment to viewing client 2504. In some embodiments, the example process may further include viewing client 2504 interpolating 2524 of the view of the content to fill views not transmitted to the viewing client. In some embodiments, the example process may further include viewing client 2504 displaying 2526 the light field for a full light field image and / or frame.

[0277] In some embodiments, the selection of a microlens representation (which may be a subsampled microlens representation) may be based on at least one of the following: the user's viewpoint, an estimated bandwidth, or the display capabilities of the viewing client. In some embodiments, the viewing client may retrieve a media manifest file describing multiple microlens representations of light field video content; and may display a set of interpolated views. In some embodiments, the viewing client may determine an estimated bandwidth between the client and the server, wherein the selected microlens representation may use the estimated bandwidth. In some embodiments, the viewing client may determine an estimated bandwidth available for streaming light field video content, such that the selected microlens representation can choose a microlens representation among the multiple microlens representations with a content size smaller than the estimated bandwidth. In some embodiments, the viewing client may track the user's gaze direction, such that the selected microlens representation can use the user's gaze direction. In some embodiments, the viewing client may estimate the bandwidth available for streaming light field video content, such that the selected subsample rate uses the estimated available bandwidth. In some embodiments, the viewing client may determine a corresponding minimum supported bandwidth for each of the multiple subsampled microlens representations. The viewing client may select a subsampled microlens representation with a maximum minimum supported bandwidth smaller than the estimated bandwidth. In some embodiments, selecting a subsampled microlens representation includes selecting a subsampled microlens representation that has a density higher than a threshold for a portion of the light field content located within a threshold of the user's gaze direction. For example, a portion of the light field image closer to the user's gaze can be represented by a subsampled representation having a higher optical density than a portion of the light field image farther from the user's gaze. In some embodiments, selecting a subsampled microlens representation includes selecting a subsampled microlens representation that has a density higher than a threshold for a portion of the light field content located within a predicted viewpoint.

[0278] In some embodiments, the viewing client can select a microlens image for each frame of each subsampled microlens representation of each part of the light field content, such that the microlens representation selected from multiple microlens representations is chosen for each part of the light field content, and the corresponding microlens image is used for interpolating the view from the subsampled microlens representation. In some embodiments, the microlens image selection can be chosen from multiple microlens images to produce the best interpolation result. In some embodiments, the microlens image selection can be based on parameters corresponding to the quality of the interpolation result from multiple microlens images. In some embodiments, the user's viewpoint can be predicted (e.g., predicted by the viewing client based on tracked viewer gaze). In some embodiments, an estimated maximum content size supported by the estimated bandwidth can be determined. A subsampled microlens representation can be selected such that it has a content size smaller than the estimated maximum content size. In some embodiments, the selected microlens representation can be adjusted based on the user's predicted viewpoint.

[0279] Figure 26 This is a flowchart illustrating an example process for a viewing client according to some embodiments. In some embodiments, the example process may include: retrieving 2602 a media manifest file describing a plurality of microlens representations of light field video content. In some embodiments, the example process may also include: estimating 2604 the bandwidth available for streaming the light field video content. In some embodiments, the example process may also include: selecting 2606 a microlens representation from the plurality of microlens representations. In some embodiments, the example process may also include: retrieving 2608 the selected subsampled representation. In some embodiments, the example process may also include: interpolating 2610 views from the microlens representation using the description of the microlens representation in the manifest file. In some embodiments, the example process may also include: displaying 2612 the interpolated views.

[0280] In some embodiments, retrieving a media manifest file may include requesting light field video content from a server. In some embodiments, retrieving a selected subsampled representation may include requesting and receiving the selected subsampled representation. In some embodiments, retrieving the subsampled representation may be done by retrieving the subsampled representation from a server. In some embodiments, another example process may include selecting a microlens representation from a plurality of microlens representations of a portion of the light field content described in a media manifest file; retrieving a subsampled microlens representation of the selected microlens representation from a server; and interpolating a view from the subsampled microlens representation. In some embodiments, an example method may include streaming a light field microlens representation of light field video content; and changing the resolution of the light field microlens representation. In some embodiments, an example method may include selecting a microlens representation from a plurality of microlens representations of a portion of the light field content; retrieving a subsampled microlens representation of the selected microlens representation; and interpolating a view from the subsampled microlens representation. In some embodiments, an apparatus may include a processor and a non-transitory computer-readable medium storing instructions that, when executed by the processor, operate to perform one or more of the methods described above.

[0281] Figure 27 This is a flowchart illustrating an example process for a viewing client according to some embodiments. In some embodiments, the example process may include receiving, 2702, a media manifest file containing multiple subsampled microlens representations describing portions of a light field video content from a server. In some embodiments, the exemplary method may also include selecting, 2704, a subsampled microlens representation from the plurality of subsampled microlens representations. In some embodiments, the example process may also include retrieving, 2706, the selected subsampled microlens representation from the server. In some embodiments, the example process may also include interpolating, 2708, a view from the retrieved selected subsampled microlens representation using a description of the selected subsampled microlens representation in the manifest file. In some embodiments, the example process may further include displaying, 2710, the interpolated view.

[0282] While methods and systems according to some embodiments have been discussed in the context of virtual reality (VR), some embodiments can also be applied to the context of mixed reality (MR) / augmented reality (AR). Furthermore, although the term "head-mounted display (HMD)" is used herein according to some embodiments, for some embodiments, some embodiments can be applied to wearable devices capable of, for example, VR, AR, and / or MR (which may or may not be attached to the head).

[0283] Example methods according to some embodiments may include: requesting light field video content from a server; receiving a media manifest file of multiple microlens representations describing multiple portions of the light field content; determining an estimated bandwidth that can be used for streaming the light field video content; selecting a subsampled microlens representation from the multiple subsampled microlens representations; requesting the selected subsampled microlens representation from the server; receiving the subsampled representation; interpolating a view from the received subsampled microlens representation using a description of the microlens representation in the manifest file; and displaying the interpolated view.

[0284] In some embodiments of the example method, the subsampled microlens representation is selected based on the estimated bandwidth.

[0285] For some embodiments of the example method, selecting a subsampled microlens representation may include: determining a corresponding minimum support bandwidth for each of the plurality of subsampled microlens representations; and selecting the subsampled microlens representation having a maximum minimum support bandwidth that is less than an estimated bandwidth among a plurality of corresponding minimum support bandwidths.

[0286] Example methods according to some embodiments may include: retrieving a media manifest file of a plurality of microlens representations describing portions of a light field video content; selecting a microlens representation from the plurality of microlens representations; retrieving the selected subsampled representation; interpolating a view from the retrieved subsampled microlens representation using a description of the microlens representation in the manifest file; and displaying the interpolated view.

[0287] Example methods according to some embodiments may include: selecting a microlens representation from a plurality of microlens representations of a portion of light field content described in a media manifest file; retrieving a subsampled microlens representation of the selected microlens representation from a server; and interpolating a view from the subsampled microlens representation.

[0288] Some embodiments of the example method may also include: retrieving a media manifest file representing a portion of the light field video content using multiple small lenses; and displaying the interpolated view.

[0289] Some embodiments of the example method may further include: determining an estimated bandwidth between the client and the server, wherein selecting the small lens indicates using the estimated bandwidth.

[0290] Some embodiments of the example method may also include: determining an estimated bandwidth that can be used to stream optical field video content; and determining an estimated maximum content size supported by the estimated bandwidth, wherein selecting the microlens means selecting a microlens representation among the plurality of microlens representations that has a smaller than the estimated maximum content size.

[0291] For some embodiments of the example method, the description of at least one of the plurality of microlens representations may include information about at least one of the range or density of the respective microlens representation.

[0292] In some embodiments of the example method, selecting the subsampled microlens representation may include selecting a subsampled microlens representation with the largest range.

[0293] In some embodiments of the example method, selecting the subsampled microlens representation may include selecting the subsampled microlens representation with the highest density.

[0294] Some embodiments of the example method may also include: tracking the user's gaze direction, wherein selecting the microlens indicates that the user's gaze direction can be used.

[0295] For some embodiments of the example method, selecting the subsampled microlens representation may include: selecting a subsampled microlens representation that has a density higher than a threshold for a portion of the light field content located within a threshold in the user's gaze direction.

[0296] In some embodiments of the example method, selecting the small lens indicates that the client's capabilities can be used.

[0297] For some embodiments of the example method, selecting the subsampled microlens representation may include selecting a subsampled microlens representation with a density higher than a threshold supported by the client.

[0298] For some embodiments of the example method, the client's capability can be the maximum microlens density supported by the client.

[0299] For some embodiments of the example method, where the interpolated view is represented from the subsampled microlens, the description of the microlens representation in the manifest file can be used.

[0300] Some embodiments of the example method may also include: updating the user's viewpoint; and adjusting the selected microlens representation for the updated viewpoint.

[0301] Some embodiments of the example method may also include: predicting the user's viewpoint; and adjusting the selected microlens representation for the predicted viewpoint.

[0302] Some embodiments of the example method may also include: selecting a subsampling rate for the selected small lens representation, wherein the selection of the subsampling rate can be made using the predicted viewpoint.

[0303] For some embodiments of the example method, selecting the subsampled microlens representation may include selecting a subsampled microlens representation that has a density above a threshold for a portion of the light field content located within a threshold at the predicted viewpoint.

[0304] Some embodiments of the example method may also include selecting a subsampling rate for the selected microlens representation.

[0305] Some embodiments of the example method may also include estimating the bandwidth available for streaming optical field video content, wherein the sub-sampling rate can be selected using the estimated available bandwidth.

[0306] Some embodiments of the example method may also include: selecting a light field spatial resolution; and dividing the light field content into multiple parts corresponding to the selected light field spatial resolution.

[0307] Some embodiments of the example method may also include adjusting the spatial resolution of the light field to improve performance metrics of the interpolated view.

[0308] Some embodiments of the example method may also include selecting a microlens image for each frame of each subsampled microlens representation for each part of the light field content, wherein selecting the microlens representation from the plurality of microlens representations may be for each part of the light field content, selecting a corresponding subsampled microlens representation, and wherein an interpolated view from the subsampled microlens representation may use the corresponding microlens image.

[0309] In some embodiments of the example method, the microlens image can be selected from a plurality of microlens images to produce the best interpolation result.

[0310] For some embodiments of the example method, interpolating a view from the retrieved subsampled microlens representation using the description of the microlens representation in the manifest file may include: unpacking the retrieved subsampled microlens representation to the original microlens position of the portion of the light field video content indicated in the manifest file; and interpolating microlens samples omitted from the retrieved subsampled microlens representation.

[0311] In some embodiments of the example method, a complete light field region image for the portion of the light field video content is generated from the retrieved subsampled small lens representation interpolated view.

[0312] Example apparatuses according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operable, when executed by the processor, to perform any of the example methods listed above.

[0313] Another example method according to some embodiments may include: streaming light field video content in a microlens representation; and changing the resolution of the light field microlens representation.

[0314] Other example methods according to some embodiments may include: selecting a microlens representation from a plurality of microlens representations of a portion of the light field content; retrieving a subsampled microlens representation of the selected microlens representation; and interpolating a view from the subsampled microlens representation to reconstruct missing microlens samples in the subsampled representation.

[0315] Another additional example method according to some embodiments may include: retrieving a subsampled microlens representation of the light field content; and reconstructing microlens samples omitted from the subsampled microlens representation by interpolating the retrieved subsampled microlens representation.

[0316] Example methods according to some embodiments may include: receiving from a server a media manifest file of a plurality of subsampled microlens representations describing portions of a light field video content; selecting a subsampled microlens representation from the plurality of subsampled microlens representations; retrieving the selected subsampled microlens representation from the server; interpolating a view from the retrieved selected subsampled microlens representation using the description of the selected subsampled microlens representation in the manifest file; and displaying the interpolated view.

[0317] Some embodiments of the example method may also include determining an estimated bandwidth that can be used to stream the light field video content.

[0318] For some embodiments of the example method, the subsampled microlens representation may be selected based on at least one of the following: the user's viewpoint, estimated bandwidth, or the display capability of the viewing client.

[0319] Some embodiments of the example method may also include predicting the user's predicted viewpoint, such that the user's viewpoint is the user's predicted viewpoint.

[0320] For some embodiments of the example method, selecting the subsampled microlens representation may include: determining a corresponding minimum support bandwidth for at least one of the plurality of subsampled microlens representations; and selecting the subsampled microlens representation having a maximum minimum support bandwidth that is less than the estimated bandwidth among a plurality of corresponding minimum support bandwidths.

[0321] Some embodiments of the example method may further include: determining the estimated maximum content size supported by the estimated bandwidth, such that selecting the subsampled microlens representation can select one of the plurality of subsampled microlens representations having a smaller than the estimated maximum content size.

[0322] Some embodiments of the example method may also include: tracking the user's gaze direction such that selecting the sub-sampling microlens represents using the user's gaze direction.

[0323] For some embodiments of the example method, the selection of a subsampled microlens representation may include selecting a portion of the light field content located within a gaze threshold in the user's gaze direction and having a density higher than a density threshold.

[0324] Some embodiments of the example method may also include: predicting the user's viewpoint; and adjusting the selected microlens representation for the predicted viewpoint.

[0325] Some embodiments of the example method may further include: selecting a light field spatial resolution; dividing the light field content into a plurality of parts corresponding to the light field spatial resolution; and selecting a microlens image for at least one frame of at least one subsampled microlens representation of at least one part of the light field content, such that selecting the subsampled microlens representation can select a corresponding subsampled microlens representation for at least one part of the light field content, and such that the corresponding microlens image can be used for interpolated views from the subsampled microlens representation.

[0326] Some embodiments of the example method may also include adjusting the spatial resolution of the light field to improve performance metrics of the interpolated view.

[0327] For some embodiments of the example method, interpolating the view from the retrieved subsampled microlens representation may include: unpacking the retrieved subsampled microlens representation to the original microlens position of the portion of the light field video content indicated in the manifest file; and interpolating microlens samples omitted from the retrieved subsampled microlens representation.

[0328] In some embodiments of the example method, the interpolated view representing the retrieved subsampled small lens can generate a complete light field region image of the portion of the light field video content.

[0329] For some embodiments of the example method, the subsampling microlens representation may be selected based on at least one of the following: the density of the selected subsampling microlens representation, or the range of the selected subsampling microlens representation.

[0330] Example apparatuses according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, operate to perform the method of any of the above-listed claims.

[0331] Example methods according to some embodiments may include: selecting a microlens representation from a plurality of microlens representations of a portion of light field content described in a media manifest file; retrieving a subsampled microlens representation of the selected microlens representation from a server; and interpolating a view from the subsampled microlens representation.

[0332] Some embodiments of the example method may also include: retrieving a media manifest file representing a portion of the light field video content using multiple small lenses; and displaying the interpolated view.

[0333] For some embodiments of the example method, the interpolated view from the retrieved subsampled microlens representation can use the description of the microlens representation in the manifest file.

[0334] Some embodiments of the example method may further include: determining an estimated bandwidth between the client and the server, such that selecting the small lens indicates that the estimated bandwidth can be used.

[0335] For some embodiments of the example method, the description of at least one of a plurality of microlens representations may include information about at least one of the range or density of the respective microlens representation.

[0336] For some embodiments of the example method, selecting a subsampled microlens representation may include selecting a subsampled microlens representation with the largest range.

[0337] For some embodiments of the example method, selecting a subsampled microlens representation may include selecting the subsampled microlens representation with the highest density.

[0338] In some embodiments of the example method, selecting a small lens indicates that the client's capabilities can be used.

[0339] For some embodiments of the example method, selecting a subsampled microlens representation may include selecting a subsampled microlens representation with a density higher than a threshold supported by the client.

[0340] In some embodiments of the example method, the capability of the client may be the maximum microlens density supported by the client.

[0341] For some embodiments of the example method, the interpolated view from the subsampled microlens representation can use the description of the microlens representation in the manifest file.

[0342] Some embodiments of the example method may also include: updating the user's viewpoint; and adjusting the selected microlens representation for the updated viewpoint.

[0343] Some embodiments of the example method may also include: predicting the user's viewpoint such that selecting a subsampled microlens representation may include selecting a portion of the light field content located within a threshold of the predicted viewpoint that has a density above the threshold.

[0344] Some embodiments of the example method may also include: selecting a subsampling rate for the selected small lens representation.

[0345] Some embodiments of the example method may further include: estimating the bandwidth available for streaming optical field video content, such that the sub-sampling rate can be selected using the estimated available bandwidth.

[0346] Some embodiments of the example method may further include: selecting a microlens image for each frame of each subsampled microlens representation of each part of the light field content, such that the microlens representation is selected from the plurality of microlens representations; selecting a corresponding subsampled microlens representation for each part of the light field content, such that the corresponding microlens image is used in the interpolated view from the subsampled microlens representation; and selecting the microlens image from the plurality of microlens images based on the estimated quality of the interpolation result.

[0347] Some embodiments of the example method may further include: determining the corresponding estimation quality of the interpolation results of the plurality of microlens images, such that the microlens image is selected based on which of the plurality of microlens images has a determined highest corresponding estimation interpolation result quality.

[0348] Example apparatuses according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operable, when executed by the processor, to perform any of the methods listed above.

[0349] Example methods according to some embodiments may include: streaming light field microlens representations of light field video content; and changing the resolution of the light field microlens representation.

[0350] Example apparatuses according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operable, when executed by the processor, to perform any of the methods listed above.

[0351] Example methods according to some embodiments may include: selecting a microlens representation from a plurality of microlens representations of a portion of the light field content; retrieving a subsampled microlens representation of the selected microlens representation; and interpolating a view from the subsampled microlens representation to reconstruct missing microlens samples in the subsampled representation.

[0352] Example apparatuses according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operable, when executed by the processor, to perform any of the methods listed above.

[0353] Example methods according to some embodiments may include: retrieving a subsampled microlens representation of the light field content; and reconstructing microlens samples omitted from the subsampled microlens representation by interpolating the retrieved subsampled microlens representation.

[0354] Example apparatuses according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operable, when executed by the processor, to perform any of the methods listed above.

[0355] Example methods according to some embodiments may include: sending a media manifest file of a plurality of subsampled microlens representations describing portions of a light field video content; receiving information indicating a subsampled microlens representation selected from the plurality of subsampled microlens representations; and sending the selected subsampled microlens representation.

[0356] Example apparatuses according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions operable, when executed by the processor, to perform any of the methods listed above.

[0357] Example apparatuses according to some embodiments may include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, operate to: transmit a media manifest file of a plurality of subsampled microlens representations describing portions of a light field video content; receive information indicating a subsampled microlens representation selected from the plurality of subsampled microlens representations; and transmit the selected subsampled microlens representation.

[0358] Note that the various hardware elements of the one or more embodiments described are referred to as “modules,” and their implementation (i.e., execution, operation, etc.) is in conjunction with the various functions described herein for each module. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices) that a person skilled in the art would consider suitable for a given implementation. Each described module may also include instructions executable to perform one or more functions described as being performed by the corresponding module, and note that these instructions may take the form of hardware (i.e., hardwired) instructions, firmware instructions, and / or software instructions, or include them, and may be stored in any suitable non-transitory computer-readable medium or media, such as commonly referred to as RAM, ROM, etc.

[0359] Although the features and elements have been described above in specific combinations, those skilled in the art will understand that each feature or element can be used alone or in any combination with other features and elements. Furthermore, the methods described herein can be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, buffer memory, semiconductor memory devices, magnetic media (e.g., internal hard disks and removable disks), magneto-optical media, and optical media (e.g., CD-ROMs and DVDs). The processor associated with the software can be used to implement radio frequency transceivers for WTRUs, UEs, terminals, base stations, RNCs, and / or any host computer.

Claims

1. A method comprising: Receive a media manifest file from the server describing regions of the light field video content. Each region is represented by a small lens and described by corresponding mapping metadata, where for each region: Each microlens representation contains microlens subviews provided at a corresponding microlens density. Using the corresponding sampling density, microlens subviews are sampled from the full array representing the region and packed into a dense array of microlens subviews. The metadata corresponding to the microlens representation is mapped, associating the position of the microlens subview in the dense array with their corresponding position in the full array of microlens subviews, and identifying the corresponding microlens density; and For each region of the light field video content: One of the small lenses representing the selected area; Retrieve the selected small lens representation from the server; Based on the corresponding mapping metadata, the retrieved small lens subviews are mapped from their positions in the dense array to their corresponding positions in the full array of small lens subviews representing the region, and the remaining small lens subviews in the full array of small lens subviews are interpolated from the mapped small lens subviews. as well as An array of subviews of small lenses.

2. The method according to claim 1, further comprising: The bandwidth available for streaming the light field video content is estimated, wherein one of the small lens representations is selected based on the estimated bandwidth.

3. The method of claim 1, wherein the selection of one of the microlens representations is based on the display capabilities of the viewing client.

4. The method according to claim 1, further comprising: Predict the viewpoint from which the user views the light field video content. One of the small lens representations is selected based on the user's predicted viewpoint.

5. The method of claim 1, wherein selecting one of the microlens representations comprises: Determine the corresponding minimum support bandwidth represented by the small lens; as well as The selection of a small lens indicates that the small lens represents the maximum minimum support bandwidth that is less than the estimated bandwidth that can be used to stream the light field video content, which is determined to be a corresponding minimum support bandwidth.

6. The method according to claim 1, further comprising: Estimate the maximum content size supported by the estimated bandwidth available for streaming the light field video content. One of the selected microlens representations includes a microlens representation with a content size smaller than the estimated maximum content size.

7. The method according to claim 1, further comprising: Track the user's gaze direction when watching light field video content. One of the microlens representations is based on the direction of the user's gaze being tracked.

8. The method of claim 7, wherein selecting one of the microlens representations comprises: Selecting a small lens indicates that it has a sampling density higher than the sampling density threshold of the region of the light field video content located within the gaze threshold of the tracked user gaze direction.

9. The method of claim 7, wherein selecting one of the microlens representations comprises: For regions of the light field video content that are closer to the tracked user's gaze direction, a microlens is selected, which has a higher microlens density than regions of the light field video content that are farther away from the tracked user's gaze direction.

10. The method according to claim 1, further comprising: Predict the viewpoint from which a user views light field video content; as well as The selected microlens representation is adjusted based on the predicted viewpoint.

11. The method according to claim 1, wherein, Based on the sampled microlens subviews, the most accurate interpolation estimate is provided for the remaining microlens subviews in the array, which are microlens subviews representing the region of light field video content sampled from the array of microlens subviews representing the region.

12. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by the processor, operate to perform the method according to any one of claims 1 to 11.

Citation Information

Patent Citations

  • Spatially tiled omnidirectional video streaming

    US11284124B2

  • Method, apparatus and stream for encoding / decoding volumetric video

    US11367247B2