Method and system for forming extended focal plane for large viewpoint variation

Abstract

Methods and systems for capturing and displaying content for a multi-focal plane (MFP) display are described. In one example, a large aperture camera is used to capture a plurality of texture images of a scene, each texture image having a different focal length. For each texture image, a focal plane image is generated. To generate focal plane images, in some embodiments, each pixel in each texture image is multiplied by a respective weight value. The weight value may be based on a measured depth of the respective pixel (e.g., as determined using a captured depth map) and / or based on a level of focus (or defocus) of the respective pixel as determined by filtering. A focal plane image may be displayed on a multi-focal plane display and may be used to generate a virtual viewpoint.

Description

[0001] This application is a divisional application of the Chinese invention patent application with the application date of June 28, 2019, application number 201980054574.8, and invention name “Method and system for forming an extended focal plane with large viewpoint changes”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application is a non-provisional application of U.S. Provisional Patent Application No. 62 / 694,722, filed on July 6, 2018, entitled “Method and System for Forming Extended Focal Planes for Large Viewpoint Changes,” and claims the benefits thereof under 35 U.S.C. §119(e), which is incorporated herein by reference in its entirety. Background Art

[0004] Forming and using multiple focal planes (MFPs) is a method for avoiding vergence-accommodation conflicts, which enables viewers to naturally focus image information along the depth dimension. This method may be particularly useful in near-eye (glasses) displays.

[0005] MFP displays create a stack of discrete focal planes, composing a 3D scene from layers along the viewer's visual axis. A view of a 3D scene is formed by projecting user-visible pixels (or voxels) at different depths and spatial angles.

[0006] Each focal plane displays a representation of the 3-D view corresponding to a portion of the depth range of the corresponding focal plane. Depth blending is a method for smoothing quantization steps and contouring when viewing views compiled from discrete focal planes, so that the user will be less likely to perceive the steps. Depth blending is described in more detail in K. Akeley et al. "A Stereo Display Prototype with Multiple Focal Distances", ACM Graphics (TOG), v.23n.3, August 2004, pp.804-813 and Hu, X., & Hua, H. (2014) "Design and assessment of a depth-fused multi-focal-plane display prototype", IEEE / OSA Journal of Display Technology, 10(4), 308-316.

[0007] When using depth blending, it has been found that rendering a relatively small number of focus planes (e.g. 4-6 planes) is sufficient for acceptable quality. This number of focus planes is also technically feasible.

[0008] A multi-focal plane display can be realized by spatially multiplexing a stack of 2-D displays or by sequentially switching the focal length of a single 2-D display in a time-multiplexed manner. The change of the focal length of a single 2-D display can be realized by high-speed birefringence (or other variable focus elements) while spatially rendering the visible portion of the corresponding multi-focal image frame. In the absence of depth mixing, it is desirable to use a greater number of focal planes, such as 14 or more, as described in JPRolland et al., "Multifocal planes head-mounted displays" Appl. Opt. 39, 3209-3215 (2000).

[0009] The human visual system (HVS) favors placing the focal plane at regular distances on a diopter scale. On the other hand, depth information is often most easily captured using a linear scale. Both options can be used for MFP displays. Examples of MFP near-eye displays are Figure 2 It is schematically shown in FIG. Figure 2A display viewed by a user's left eye 202 and right eye 204 is shown. A respective eyepiece 206, 208 is provided for each eye. The eyepiece focuses an image formed by a respective image stack 210, 212. The image stack forms different images at different distances from the eyepiece. To the user's eyes, the images appear to originate from different virtual image planes, such as image planes 214, 216, 218.

[0010] The MFP display creates an approximation of the light field of the displayed scene. Because the near-eye display moves with the user's head movement, it is sufficient to support only one viewpoint at each moment. Accordingly, the approximation of the light field is easier because there is no need to capture the light field of a large number of viewpoints. Summary of the invention

[0011] The present disclosure describes methods and systems for capturing and displaying content for a multi-focal plane (MFP) display. In some embodiments, content is generated from a focal stack (images captured at varying focal lengths). Some embodiments can produce a reduced amount of de-occlusion and holes when moving the MFP for large synthetic parallax or viewpoint changes.

[0012] In some embodiments, the focused image is captured with a large aperture so that some image information is obtained from behind occluding objects.

[0013] Some embodiments also perform large aperture depth sensing, which can be achieved by a large aperture depth sensor, by applying a defocus map, or by using a suitable filtering and redistribution scheme on the focus stack and / or focus plane formed thereby. In some embodiments, filtering is applied to focus the stack image before forming the redistributed focus plane. In some embodiments, filtering is applied after forming the focus plane. The filtering results are then used to form the redistributed focus plane (or more generally, the high frequency and / or redistributed focus plane).

[0014] An example operation is as follows: Get multiple texture images p of the scene i , where each texture image has a different corresponding focal length d i The texture image can be, for example, an RGB image or a grayscale image, among other options. For each texture image p i , generating the focal plane image q i To generate the focal plane image q i , texture image p i Each pixel (x, y) in i (x,y) weighted. The texture image p i Each pixel value p i (x,y) multiplied by their respective weights w i (x,y) to generate the focal plane image qi , so that q i (x,y)=p i (x,y)·w i (x,y).

[0015] Weight w i (x,y) can be represented in the texture image p i The amount of focus at a pixel (x,y) in the texture image p can be determined using different techniques. i In some such techniques, the depth z of the pixel (x, y) is measured or otherwise determined. i (x,y), and weight w i (x,y) is a function of depth such that w i (x,y)=w i [z i (x,y)]. Function w i [z] may be a blending function as used in known multi-focal displays. In some embodiments, the function is i [d i ] has a maximum value (e.g., a value of 1), indicating that the pixel is most likely in focus when its measured depth is the same as the focal length. i The value of [z] can be changed as z changes from the focal length d i Increases or decreases monotonically, giving lower weights to pixel depths that are further away from the focal distance and less likely to be in focus. Pixels with depth values ​​that are sufficiently offset from the focal plane may be given a weight of zero (even if some level of focus can be discerned).

[0016] In some embodiments, in the texture image p i The amount of focus at a pixel (x, y) is determined by generating a defocus map, which is a texture image p i Each pixel in the image is assigned a focus level (or defocus level). For example, the most focused pixel can be given a weight of one, and the less focused pixels can be given weights as low as zero.

[0017] N focal plane images q o …q i …q N-1 A collection of can be generated using the techniques described here and can be displayed on a multiple focal plane display. Depending on the type of display, the focal plane images can be displayed simultaneously or in a rapidly cycled sequence using time multiplexing.

[0018] In some embodiments, the texture image p iThe set of may be greater than the number of available (or desired) display planes in the multi-focal plane display. In this case, a method may include selecting a focal plane image for each display plane. For each display plane, a texture image having a focal length that is the same as or closest to the focal length of the display plane may be selected.

[0019] In some embodiments, a virtual viewpoint is generated by laterally shifting at least a first one of the focus plane images relative to at least a second one of the focus plane images. For example, the focus plane images may be laterally shifted by an amount inversely proportional to the display focal length of the corresponding focus plane image (i.e., the focal length of the display plane of the focus plane image). The virtual viewpoint may be used as one or both of a stereoscopic viewpoint pair. The virtual viewpoint may also be generated in response to viewer head motion to simulate motion parallax.

[0020] In some embodiments, each texture image p is captured substantially simultaneously. i and the corresponding depth map d i Each texture image and the corresponding corresponding depth map may be captured using the same or similar optics. Each texture image and the corresponding corresponding depth map may be captured using optics having the same aperture. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1A is a system diagram illustrating an example communication system in which one or more disclosed embodiments may be implemented.

[0022] Figure 1B is a diagram showing that according to the embodiment, Figure 1A A system diagram of an example wireless transmit / receive unit (WTRU) for use in a communication system is shown.

[0023] Figure 2 is a schematic diagram of a multifocal near-eye display that may be used in some embodiments.

[0024] Figures 3A-3D is a schematic representation of different texture images of the same scene. Figure 3A is a schematic representation of an extended depth of field image of a scene. Figure 3B-3D is a schematic representation of images of a scene captured with a larger aperture camera focused at different distances.

[0025] Figures 4A-4D is a schematic diagram representing depth maps captured at different focal lengths. Figures 4A-4D The schematic depth map shown in corresponds to Figures 3A-3D The corresponding texture image shown in .

[0026] Figures 5A-5C According to Figures 4B-4D Schematic diagram of the focus weight map generated from the depth map.

[0027] Fig. 6A Schematically shows the Figure 3B Texture images and Figure 5A The focus weight map generates a focal plane image.

[0028] Figure 6B Schematically shows the Figure 3C Texture images and Figure 5B The focus weight map generates a focal plane image.

[0029] Figure 6C Schematically shows the Figure 3D Texture images and Figure 5C The focus weight map generates a focal plane image.

[0030] Figure 7 Schematically showing the display to the user Figures 6A-6C focal plane image.

[0031] Figure 8 is a flow chart illustrating a method of generating and displaying a focal plane image in some embodiments.

[0032] Fig. 9 is a schematic diagram of different focal lengths encountered in some embodiments.

[0033] Fig.10 is a flow chart illustrating a method of generating and displaying a focal plane image in some embodiments.

[0034] Fig.11 is a flow chart illustrating a method of generating and displaying a focal plane image in some embodiments.

[0035] Fig.12 is a flow chart illustrating a method of generating and displaying a focal plane image in some embodiments.

[0036] Figures 13A-13C In some embodiments, the depth z is shown for different focal planes. i The focus weight w of the function of (x,y) i Example of (x,y).

[0037] Figures 14A-14C In an additional embodiment, the depth z is shown for different focal planes. i The focus weight w of the function of (x,y) i Example of (x,y).

[0038] Example network for implementation of embodiments

[0039] Figure 1A1 is a diagram illustrating an example communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content such as voice, data, video, messaging, broadcast, etc. to multiple wireless users. The communication system 100 may enable multiple wireless users to access such content by sharing system resources including wireless bandwidth. For example, the communication system 100 may use one or more channel access methods such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single carrier FDMA (SC-FDMA), zero tail unique word DFT spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block filtered OFDM, and filter bank multi-carrier (FBMC), etc.

[0040] like Figure 1A As shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106 / 115, public switched telephone network (PSTN) 108, Internet 110 and other networks 112, however it should be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each WTRU 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. For example, any of the WTRUs 102a, 102b, 102c, 102d may be referred to as a “station” and / or “STA”, which may be configured to transmit and / or receive wireless signals, and may include a user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular phone, a personal digital assistant (PDA), a smart phone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (IoT) device, a watch or other wearable device, a head-mounted display (HMD), a vehicle, a drone, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in an industrial and / or automated process chain environment), consumer electronic devices, and devices operating on a commercial and / or industrial wireless network, etc. Any of the WTRUs 102a, 102b, 102c, 102d may be interchangeably referred to as a UE.

[0041] The communication system 100 may also include a base station 114a and / or a base station 114b. Each base station 114a, 114b may be any type of device configured to facilitate access to one or more communication networks (e.g., CN 106 / 115, Internet 110, and / or other networks 112) by wirelessly interfacing with at least one of the WTRUs 102a, 102b, 102c, 102d. For example, the base stations 114a, 114b may be base transceiver stations (BTS), Node Bs, eNode Bs, Home Node Bs, Home eNode Bs, gNBs, NR Node Bs, site controllers, access points (APs), wireless routers, and the like. Although each base station 114a, 114b is depicted as a single element, it should be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0042] The base station 114a may be part of the RAN 104 / 113, and the RAN may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), a relay node, and the like. The base station 114a and / or the base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies called cells (not shown). These frequencies may be in a licensed spectrum, an unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide wireless service coverage for a specific geographic area that is relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, a cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e., one transceiver corresponds to one sector of the cell. In an embodiment, the base station 114a may use multiple-input multiple-output (MIMO) technology, and may use multiple transceivers for each sector of the cell. For example, by using beamforming, signals may be transmitted and / or received in a desired spatial direction.

[0043] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, millimeter wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0044] More specifically, as described above, the communication system 100 may be a multiple access system and may use one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, and SC-FDMA, among others. For example, the base station 114a in the RAN 104 / 113 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may use Wideband CDMA (WCDMA) to establish the air interface 115 / 116 / 117. WCDMA may include communication protocols such as High Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High Speed ​​UL Packet Access (HSUPA).

[0045] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or Advanced LTE (LTE-A) and / or Advanced LTE Pro (LTE-A Pro).

[0046] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as NR radio access, which may establish the air interface 116 using New Radio (NR).

[0047] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may jointly implement LTE radio access and NR radio access (e.g., using dual connectivity (DC) principles). Thus, the air interface used by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., eNBs and gNBs).

[0048] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi)), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data Rates for GSM Evolution (EDGE), and GSM EDGE (GERAN), among others.

[0049] Figure 1A The base station 114b in the example can be, for example, a wireless router, a Home NodeB, a Home eNodeB, or an access point, and can use any suitable RAT to facilitate wireless connectivity in a local area, such as a business location, a residence, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a road, and the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d can establish a wireless local area network (WLAN) by implementing a radio technology such as IEEE 802.11. In an embodiment, the base station 114b and the WTRUs 102c, 102d can establish a wireless personal area network (WPAN) by implementing a radio technology such as IEEE 802.15. In another embodiment, the base station 114b and the WTRUs 102c, 102d can establish a picocell or a femtocell by using a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.). Figure 1A As shown, the base station 114b may be directly connected to the Internet 110. Thus, the base station 114b does not need to access the Internet 110 via the CN 106 / 115.

[0050] The RAN 104 / 113 may be in communication with the CN 106 / 115, which may be any type of network configured to provide voice, data, applications and / or Voice over Internet Protocol (VoIP) services to one or more WTRUs 102a, 102b, 102c, 102d. The data may have different quality of service (QoS) requirements, such as different throughput requirements, latency requirements, fault tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. The CN 106 / 115 may provide call control, billing services, mobile location-based services, prepaid calls, Internet connectivity, video distribution, etc., and / or may perform advanced security functions such as user authentication. Although in Figure 1ANot shown, however, it will be appreciated that the RAN 104 / 113 and / or the CN 106 / 115 may be in direct or indirect communication with other RANs that employ the same RAT or a different RAT as the RAN 104 / 113. For example, in addition to being connected to the RAN 104 / 113, which may employ NR radio technology, the CN 106 / 115 may also be in communication with another RAN (not shown) employing GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0051] The CN 106 / 115 may also act as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network that provides plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer network devices that use common communication protocols, such as TCP, User Datagram Protocol (UDP), and / or IP in the Transmission Control Protocol / Internet Protocol (TCP / IP) Internet protocol suite. The networks 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs, where the one or more RANs may use the same RAT or a different RAT as the RAN 104 / 113.

[0052] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communication system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links). Figure 1A The illustrated WTRU 102c may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[0053] Figure 1B is a system diagram illustrating an example WTRU 102. Figure 1B As shown, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a numeric keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138. It should be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0054] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal decoding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. Although Figure 1B The processor 118 and the transceiver 120 are depicted as separate components, however it should be appreciated that the processor 118 and the transceiver 120 may also be integrated together in an electronic package or chip.

[0055] The transmit / receive element 122 may be configured to transmit or receive signals to or from a base station (e.g., base station 114a) via the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. As an example, in an embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive RF and light signals. It should be appreciated that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0056] Although in Figure 1B 102 as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0057] The transceiver 120 may be configured to modulate signals to be transmitted by the transmit / receive element 122 and to demodulate signals received by the transmit / receive element 122. As described above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers that allow the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0058] The processor 118 of the WTRU 102 may be coupled to a speaker / microphone 124, a numeric keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit), and may receive user input data from these elements. The processor 118 may also output user data to the speaker / microphone 124, the numeric keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from and store data in any suitable memory such as a non-removable memory 130 and / or a removable memory 132. The non-removable memory 130 may include a random access memory (RAM), a read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, and a secure digital (SD) memory card, etc. In other embodiments, the processor 118 may access information from, and store data in, memory that is not physically located on the WTRU 102, such as, for example, on a server or a home computer (not shown).

[0059] The processor 118 may receive power from the power source 134, and may be configured to distribute and / or control power for use by other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (Ni-Cd), nickel-zinc (Ni-Zn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0060] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) related to the current location of the WTRU 102. In addition to or in lieu of the information from the GPS chipset 136, the WTRU 102 may receive location information from a base station (e.g., base stations 114a, 114b) via the air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It should be appreciated that the WTRU 102 may acquire location information via any suitable location-determination method while remaining consistent with an embodiment.

[0061] The processor 118 may be further coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, module, a frequency modulation (FM) radio unit, a digital music player, a media player, a video game console module, an Internet browser, a virtual reality and / or augmented reality (VR / AR) device, and an activity tracker, etc. The peripheral device 138 may include one or more sensors, which may be one or more of the following: a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor, a geolocation sensor, an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a posture sensor, a biometric sensor, and / or a humidity sensor.

[0062] The WTRU 102 may include a full-duplex radio for which reception and transmission of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent or simultaneous, etc. A full-duplex radio may include an interference management unit that reduces and / or substantially eliminates self-interference by means of hardware (e.g., chokes) or by signal processing by a processor (e.g., a separate processor (not shown) or by the processor 118). In an embodiment, the WTRU 102 may include a half-duplex radio that transmits or receives some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) or downlink (e.g., for reception)).

[0063] Despite Figure 1A-1B While the WTRU is described as a wireless terminal, it is contemplated that in certain representative embodiments such a terminal may utilize (eg, temporarily or permanently) a wired communication interface with a communication network.

[0064] In a representative embodiment, other network 112 may be a WLAN.

[0065] In view of Figure 1A-1B and about Figure 1A-1B One or more or all of the functions described herein may be performed by one or more simulation devices (not shown). A simulation device may be one or more devices configured to simulate one or more or all of the functions described herein. For example, a simulation device may be used to test other devices and / or simulate network and / or WTRU functions.

[0066] The simulation device can be designed to implement one or more tests about other devices in a laboratory environment and / or an operator network environment. For example, the one or more simulation devices can perform one or more or all functions while being implemented and / or deployed as part of a wired and / or wireless communication network in whole or in part to test other devices inside the communication network. The one or more simulation devices can perform one or more or all functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. The simulation device can be directly coupled to other devices to perform the test, and / or can use over-the-air wireless communication to perform the test.

[0067] One or more simulation devices can perform one or more functions, including all functions, while not being implemented or deployed as part of a wired and / or wireless communication network. For example, the simulation device can be used in a test lab and / or a test scenario of a wired and / or wireless communication network that is not deployed (e.g., testing) to implement tests on one or more components. The one or more simulation devices can be test devices. The simulation device can transmit and / or receive data using direct RF coupling and / or wireless communication with the aid of RF circuits (as an example, the circuits can include one or more antennas). DETAILED DESCRIPTION

[0068] The impact of using a wide aperture in image capture.

[0069] Real cameras using a finite aperture produce images with a certain depth of field (DoF). Depth of field can be described as the span of distances from the capture point within which pixels are in focus. Outside the DoF, pixels become defocused or blurred.

[0070] When the camera parameters are known, the DoF can be calculated or estimated using known formulas.

[0071] At one extreme, an ideal pinhole camera is a camera with an infinitely small aperture. An ideal pinhole camera produces an image with infinite DoF, and all pixels are in focus regardless of their depth. In practice, under very good lighting conditions, a pinhole camera can be approximated by using a small aperture in a physical camera.

[0072] Under practical imaging conditions, an approximation to a pinhole image can be achieved by capturing and combining focus stacked images, images with several focal lengths. Various algorithms exist to combine these images into one extended focus image. An extended focus image is formed using a discrete set of focus captures.

[0073] Figure 3Ais a schematic representation of a pinhole image 302 of a scene including a nearby radio, a window frame at an intermediate distance, and buildings at a greater distance. Due to the small aperture of the pinhole camera, objects at all distances are in focus.

[0074] Figure 3B-3D is a schematic representation of images of the same scene captured with a larger aperture camera focused at different distances. In some embodiments, Figure 3B-3D The images are captured simultaneously or substantially simultaneously (or, in the case of virtual images, generated to represent substantially the same moment in time).

[0075] Figure 3B Is focused at close distance f 1 Schematic representation of an image 304 of FIG. The radio device itself is in focus (shown with solid lines), while the more distant window frames and buildings are at least partially out of focus (shown with dashed lines).

[0076] Figure 3C is focused at the middle distance f 2 306. The window frame is in focus, but the nearby radio equipment and the more distant buildings are at least partially out of focus. Figure 3C As also schematically shown in , given a non-zero aperture size, the focus window is "visible through" the portion of the radio's antenna that is out of focus. While this effect is most noticeable in narrow objects (such as a radio antenna), it is also noticeable around the edges of larger real-world objects. However, to maintain clarity and reproducibility of this figure, this effect is only shown here for narrow objects.

[0077] Figure 3D Is focused at a greater distance f 3 308 is a schematic representation of an image of a radio device. The nearby radio equipment and the slightly more distant window frame are out of focus, but the building visible through the window is in focus. Given the aperture size, the narrow muntins (horizontal and vertical crossbars) of the window frame appear at least partially transparent, as do the radio antennas. As a result, the radio antennas are not as transparent as those in the image above. Figure 3A Compared with the pinhole image, the captured image contains information about the building on a larger scale.

[0078] Figures 3A-3D The effect of occlusion in different focus captures is shown. Figures 3A-3D shows that the occlusion caused by close objects does not remain the same when changing the focal length. Figure 3D In the image, with the focus on distant buildings, occlusions such as window panes and radio antennas appear eroded and filtered, making them look different from the image with the focus on closer distances. Figure 3B and 3C Compared to the image above, it reveals more background details.

[0079] Such as Figures 3A-3D Those images represented may contain information such as color and brightness information (e.g., luminance and chrominance information or RGB information) but not depth information, and are referred to herein as texture images. In some cases, where image data includes both texture and depth information (e.g., RGBD images), the term texture image may be used to refer to a portion of the image data that contains brightness and / or color information.

[0080] Depth measurement technology.

[0081] Some types of depth sensors use conventional camera optics and produce a depth map that is similar to a photograph by its focus properties. Typically, for depth sensing, a small aperture is advantageous in order to obtain a depth map that is focused over a large depth of field (DoF). A large aperture will increase sensitivity and range, but will also reduce DoF.

[0082] An example of a depth sensor system with a relatively large aperture is described in S. Honnungar et al., "Focal-sweep for Large Aperture Time-of-Flight Cameras", IEEE International Conference on Image Processing (ICIP), 2016, pp. 953-957. In some embodiments, such a large aperture time-of-flight camera can be used for depth sensing.

[0083] An example of a device capable of generating a depth map (indicating pixel distance from the capture device) is the Kinect sensor. The depth map can be used when the view is resolved into focal planes (MFP). An alternative technique is to take a camera-based focal capture and use filtering and other image processing means to derive a depth map using a "depth from focal distance" method.

[0084] One property of a defocus map is that it shows objects with the same defocus value despite being at the same distance behind or in front of the focus. Another property is that defocus map values ​​- although nonlinear with respect to depth - can be mapped to linear distances by using information about camera parameters (aperture, focal length, etc.), as described in Shaojie Zhuo, TerenceSim, "Defocus map estimation from a single image", Pattern Recognition 44 (2011), pp. 1852-1858.

[0085] Problems that are solved in some embodiments.

[0086] Multi-focal plane (MFP) representations offer the benefit of supporting viewer accommodation without the extreme bandwidth and challenge of capturing a full light field representation. A limitation of current MFP methods is that the information present in the entire light field cannot be fully preserved due to information loss caused by, for example, occlusion.

[0087] Existing MFP methods usually use a texture image and the corresponding depth map as input. The accuracy with which each texture image is acquired, in addition to several other parameters affecting the quality, limits the quality of the corresponding MFP decomposition process and its result, the focal plane.

[0088] Furthermore, current methods generally do not exploit the additional information provided by a focus stack, which is a collection of images captured from one view at varying focal lengths. In particular, current methods generally do not exploit the additional information provided by a focus stack captured using a large aperture. This results in a loss of information that might otherwise be captured behind or through an occluding object or structure when captured using a large aperture.

[0089] In traditional MFP methods, a depth map is formed based on a "pinhole viewpoint" and the same segmentation (occlusion) is used when forming the MFP at each distance. In order to capture more information from the scene, some examples described herein use several focus captures (called a focus stack) and a separate depth-based segmentation (depth map) for each captured image.

[0090] Forming and using MFP is a method for avoiding vergence-accommodation conflicts so that the viewer can naturally focus on image information in the depth dimension. The method is particularly useful in near-eye (glasses) displays. It was found that rendering a relatively small number of MFPs (4-6) is sufficient for quality while being technically feasible.

[0091] In current methods for MFP formation, texture images and corresponding pixel distances (depth maps) are typically used. In some cases, this information is virtual and generated using 3D modeling, resulting in a texture that is in focus everywhere (called fully focused content).

[0092] 3D information can also be captured from a real-world view. This view can be captured by a physical camera with a focal length, aperture, and other parameters, which results in a texture image that is only in focus at a certain distance from the capture device. Accordingly, the content is not fully in focus.

[0093] Overview of embodiments.

[0094] The process and system described herein operate to form multiple focal planes (MFP) using a focal stack (image with varying focal length) as input. In one example, multiple conventional MFP formation processes are performed in parallel for each focal stack image, and the pixel and depth with the best focus are used.

[0095] Capturing a scene at varying focal lengths may also be applied to depth sensors, which in some embodiments use relatively large aperture optics with variable focal length.

[0096] The depth-based decomposition uses a different segmentation (depth map) for each texture image. Accordingly, in some embodiments, the resulting MFP uses all focal stack images and (most of) their information content. Specifically, more information is captured into the focal plane around the occlusion compared to the conventional MFP approach.

[0097] In general, a larger aperture results in more image information being captured after occluding the edges of an object. This information expands the focal plane images and creates some overlap between them. This overlap can appear as some brightening near the edges of the object when the focal plane images are superimposed. Depending on the desired use, there may be an optimal amount of overlap with respect to perceived image quality. Accordingly, the aperture size may be selected to be sufficient to capture enough of the occluded area without overly highlighting or enhancing the edges of the object.

[0098] In some embodiments, due to the use of a relatively large aperture when capturing multiple focus stack images, information behind occluded objects or image areas is also captured and passed to the MFP formation process. Unlike when combining the focus stack images into one extended focus image, this additional information is preserved in the process and results in an MFP stack with an extended amount of information, referred to herein as an extended MFP stack with an extended focal plane.

[0099] Some embodiments use a focal stack (a series of texture images) and a series of corresponding depth maps as input. A focal stack image may be captured by taking a series of texture images at different focal lengths or resolving the texture images from a light field captured from the view. The series of texture images and corresponding depth maps are sent after applying an appropriate compression scheme to the data.

[0100] In conventional MFP processing, a single texture image is multiplied by a focus weight map derived from a single depth map. In some embodiments, on the other hand, a series of texture images captured at different focal lengths corresponds to a series of slightly different depth maps and focus weight maps. The depth maps are captured using a relatively large aperture and varying focal lengths. In some embodiments, the same aperture and focal length (optics) are used for the texture images in the focus stack.

[0101] The received depth map is used to generate a focus weight map, which is used for the formation and blending of the focus plane (MFP). Each texture image in the focus stack is multiplied with a corresponding focus weight map to form a corresponding focus plane image. In some embodiments, each texture image is multiplied with a focus weight map formed from a depth map captured according to / for the same focal length.

[0102] Conventional MFP methods decompose one image with one focal length or one extended focus image (virtual modeling scene or compilation of several texture images).A considerable amount of information behind occluded objects or areas does not enter the MFP formation process.

[0103] In some embodiments, each focal plane image is formed using its corresponding focus capture as input. In addition to collecting accurate information from all focal lengths, the method also exploits information behind occluded objects or areas. The focal plane generated using the techniques described herein is called an extended MFP.

[0104] Embodiments used herein can be employed in systems that use focal planes to generate virtual viewpoint changes. Generation of virtual viewpoint changes can be performed for laterally displaced viewpoints by displacing the MFPs laterally relative to each other. The amount of displacement depends on the selected viewpoint change amount (parallax) and the distance of each MFP from the viewer. In some embodiments, generation of virtual viewpoint changes can be performed for viewpoints displaced in a forward or backward direction by scaling of the MFPs, where closer MFPs are scaled by a larger amount than farther MFPs.

[0105] Compared with the shift of the traditional MFP, the shift of the extended MFP can result in a reduced level of de-occlusion or holes. Accordingly, this benefit can be used to increase the amount of parallax of the virtual viewpoint change.

[0106] Some existing methods use focus capture as input to the MFP decomposition procedure, but have been limited to aperture sizes typical for the human eye (on the order of 3-4 mm under normal viewing conditions). These methods cannot be used to exploit the internal painting effect (hole reduction) achieved by extending the focal plane behind the occluding object.

[0107] Some embodiments benefit from using a large aperture when capturing focus stacks. Light fields are also a viable option for providing large aperture images with varying focal lengths. Unlike light field solutions that are based on transmitting all captured data, some embodiments produce an MFP that operates as an approximation of a light field, which can be efficiently compressed and transmitted.

[0108] Due to acceptable technical complexity and good rendering quality, MFP displays are a very viable approach for supporting natural adaptation of 3D content. Therefore, using MFP is also a very natural choice for capture and transmission support.

[0109] Filtering based embodiments may operate to capture focus properties also for possible non-Lambertian phenomena in the scene (eg, also showing correct focus for reflected and refracted image information).

[0110] Example image processing and display methods.

[0111] A set of N images with different focal lengths is captured.

[0112] Some embodiments utilize additional information obtained from the scene when using relatively large aperture image capture. Relatively large aperture here means substantially larger than the aperture of the human eye, which is about 3 mm under normal conditions. For example, an aperture of 1 cm or larger can be used. In some embodiments, an aperture of about 36 mm can be used. In some embodiments, the aperture is in the range of one centimeter to several centimeters.

[0113] Capture a set of N texture images of the scene, with focal length f 1 ,f 2 …f N .For example, Figure 3B , 3C and 3D texture images can be represented by the corresponding focal length f 1 、f 2 、f 3 In this example, N is the same relatively small number of focal planes used in both MFP formation and rendering. For simplicity, in this example, the MFP is shown at the corresponding distance d i Where d 1 =f 1 ,d 2 =f 2 …d N =f N .

[0114] When changing the focal length, the obtained texture image is focused at the corresponding distance. Due to the large aperture used, each texture image can also contain some information from behind the edges of occluding objects, such as Figure 3D The part of the building that is occluded by the foreground object. Later in the process, this information finally reaches the corresponding focal plane (MFP). In some examples, the symbol p i Used to refer to a texture image indexed by the value i, where p i (x,y) is the pixel value (eg, brightness value or RGB value, among other possibilities) at location (x,y) within the texture image.

[0115] N depth maps are captured or formed.

[0116] In this example, for each of the N texture images, a separate depth map is captured.In case of different focal lengths, the optimal segmentation of the scene and the corresponding allocation of depth pixels may be different.

[0117] In some examples, the symbol z i Used to refer to texture images i In some embodiments, the depth map corresponding to the texture image p is used. i The focal length is the same as the focal length d i To capture the depth map z i Among other options, a time-of-flight camera or a structured light camera can be used to capture the depth map. Symbol z i (x,y) is used in some examples to refer to the depth recorded for the location (x,y) within the texture image.

[0118] In a depth map captured with a large aperture size (e.g., 1 cm or larger), the boundary between closer objects and farther objects may be "blurred." For example, a depth map captured with a large aperture may show a gradual transition in measured distance across pixels, even if there is actually a sharp boundary between the closer and farther objects. For example, in the case of a time-of-flight camera such as used by Honnungar et al., pixels near the boundary may measure a superposition of time-modulated light, thereby combining light reflected from closer objects with light reflected from farther objects. When the received light is processed to measure the "time of flight" (e.g., according to Equation 1 of Honnungar et al.), the result may reflect a depth between the depth of the closer object and the depth of the farther object. Although this "blurring" of depth values ​​may be considered undesirable in existing systems, this effect is used in some examples described herein to advantageously form an extended focal plane for display while reducing the appearance of holes or gaps between focal planes.

[0119] The focal length of the depth detection optics is adjusted so that each of the N depth maps is in focus at the same distance as the corresponding focus captured image. When a large aperture is used, depth values ​​for pixels / areas occluded by closer objects can also be obtained.

[0120] Figures 4A-4D is a schematic diagram showing depth maps captured with different focal lengths. Figures 4A-4D The schematic depth map shown in corresponds to Figures 3A-3D The corresponding texture image shown in .

[0121] Figure 4A represents images captured with a very small aperture and essentially infinite depth of focus. Figures 3A-3D A "pinhole" depth map 402 of a scene of FIG. Further areas are represented by darker shading, while closer areas are represented by lighter shading (or no shading for the closest areas).

[0122] Figure 4B Is to use Figure 3B The same aperture and focal length f used to capture the texture image 1 Schematic diagram of the captured depth map 404. Specifically, Figure 4B The depth map of is focused on the radio in the foreground. The window panes further away are out of focus and therefore appear at least partially transparent to the depth camera. (For illustration, Figure 4B The panes in the schematic diagram are treated as completely transparent).

[0123] Figure 4C Is to use Figure 3C The same aperture and focal length f used to capture the texture image 2 Schematic diagram of the captured depth map 406. Specifically, Figure 4C The depth map of is focused on the window frame at an intermediate distance. The radio is closer to the camera than the focal length. As a result, the radio antenna appears at least partially transparent to the depth camera. (For illustration, in Figure 4C In the schematic diagram, the antenna is treated as completely transparent).

[0124] Figure 4D Is to use Figure 4D The same aperture and focal length f used to capture the texture image 3 Schematic diagram of the captured depth map 408. Specifically, Figure 4D The depth map of the image is focused on the building in the background. The radio and window frame are closer to the camera than the focal length. As a result, the radio antenna and window slats appear at least partially transparent to the depth camera. (For illustration, the antenna and slats are shown in the image above.) Figure 4D is treated as completely transparent in the schematic diagram)

[0125] Generate N focus weight maps.

[0126] Depth blending may be achieved by applying a depth blending function to the depth map, for example, as described in Kurt Akeley, Simon J. Watt, Ahna Reza Girshick, and Martin S. Banks (2004), "A Stereo Display Prototype with Multiple Focal Distances," ACM Graphics Processing (TOG), v. 23n. 3, August 2004, pp. 804-813. In some embodiments, a linear filter (also called a tent filter) is used, although in some embodiments a non-linear filter may be used.

[0127] In some embodiments, the depth map is used to generate a focus weight map (eg, N focus weight maps) that indicates the weight that an image pixel contributes to each focus plane image.

[0128] In some such embodiments, those pixels that are exactly at the distance of a focal plane contribute only to the corresponding focal plane (with full weight w=1). Due to depth blending, pixels between two focal planes contribute to both planes with the weights (w1 and w2; w1+w2=1) represented by the corresponding focus weight map.

[0129] Symbol w j (x, y) can be used to represent the focus weight of a pixel at position (x, y) relative to the display focus plane indexed by j. In some examples, the focus weight map w j (x,y) is a function of depth such that w j (x,y)=w j [z i (x,y)], where z i (x,y) is the depth of the pixel at position (x,y) in the depth map indexed by i (corresponding to the texture image indexed by i).

[0130] In some embodiments, each of the N depth maps corresponding to the N images is processed by N blending functions. Thus, a total of N×N focus weight maps may be generated, where in some examples each focus weight map may be composed of w ij (x,y)=w j [z i (x, y)], where i, j = 0, ..., N-1. A feasible option is to use only those focus weight maps corresponding to the focal length of each texture image, so that in such an embodiment each focus weight map can be represented by w j (x,y)=w j [z i(x, y)]. Each such focus weight map contains the best focused and accurate information compared to any other focus weight map. In alternative embodiments, for example to provide a desired visual effect, one or more focus weight maps may be selected that do not correspond to the focal length of the texture image.

[0131] Figures 5A-5C is with Figure 3B-3D The corresponding texture images of are used together to generate the distance f 1 、f 2 、f 3 Schematic diagram of a focus weight map for a corresponding focal plane image of . The focus weight map provides a weight that represents, for each region (e.g., pixel) of the texture image, the amount that the region is in focus. Figures 5A-5C In the illustration of , areas with higher weights (corresponding to more in-focus areas) are shown with lighter (or absent) shading, while areas with lower weights (corresponding to more out-of-focus areas) are shown with darker shading.

[0132] Figure 5A Schematically shows the generation distance f 1 The focus weight map 504 used in the focal plane image of . 1 The area of ​​depth is given the highest focus weight, such as using Figure 4B The depth map is measured at the focal length f 1 For example, since the radio equipment is basically positioned at a distance f 1 The radio is at a distance from the camera, so the radio is in focus, and the area (e.g., pixel) corresponding to the radio is given the highest focus weight. Other areas such as window frames and background buildings are at a greater distance, are more out of focus, and therefore have a lower focus weight.

[0133] Figure 5B Schematically shows the generation distance f 2 The focus weight map 506 used in the focal plane image of . 2 The area of ​​depth is given the highest focus weight, such as using Figure 4C The depth map is measured at the focal length f 2 For example, because the window frame is basically positioned at a distance f 2 The window frame is in focus, and the area (e.g., pixel) corresponding to the window frame is given the highest focus weight. The radio is out of focus because it is closer than the distance f. 2 is closer, and the background building is out of focus because it is smaller than f 2 are at a greater distance, so those areas have a lower focus weight.

[0134] Figure 5C Schematically shows the generation distance f 3 The focus weight map 508 used in the focal plane image of . 3 The area of ​​depth is given the highest focus weight, such as using Figure 4D The depth map is measured at the focal length f 3 For example, because the background buildings are basically located at a distance f 3 The building is at a closer distance, so the building is in focus, and the area (e.g., pixel) corresponding to the building is given the highest focus weight. Other areas such as window frames and radio equipment are at a closer distance, are more out of focus, and therefore have a lower focus weight.

[0135] Selection and use of N focal plane images.

[0136] In some embodiments, a focus plane image is formed by multiplying each texture image by a focus weight map corresponding to its focal length. Formed in this way, the focus plane also contains some information behind the edges of occluding objects. When capturing a focused image (and sensing depth), the larger the aperture, the greater the amount of this information.

[0137] Fig. 6A Schematically shows the focal length f 1 A focus plane image 604 is generated for display at focal length f. In this example, the focus plane image 604 is generated by multiplying the texture image 304 by the focus weight map 504 on a pixel-by-pixel basis (possibly after scaling or otherwise aligning the texture image and the focus weight map). The most visible content in the focus plane image 604 is primarily the radio device, which is at focal length f. 1 The object that is in most focus.

[0138] Figure 6B Schematically shows the focal length f 2 A focus plane image 606 is generated for display at focal length f. In this example, the focus plane image 606 is generated by multiplying the texture image 306 by the focus weight map 506. The most visible content in the focus plane image 606 is mainly the window frame, which is at focal length f. 2 The object that is in most focus.

[0139] Figure 6C Schematically shows the focal length f 3 A focus plane image 608 is generated for display at focal length f. In this example, the focus plane image 608 is generated by multiplying the texture image 308 by the focus weight map 508. The most visible content in the focus plane image 608 is mainly the building, which is at focal length f. 3 The object that is in most focus.

[0140] Figure 7 Schematically showing, for example, the use of Figure 2 The MFP display of FIG. 600 displays multiple focal plane images to the user. Specifically, in this example, focal plane image 604 is displayed at the focal plane closest to the user (left side of the figure), focal plane image 606 is displayed at a focal plane farther from the user, and focal plane image 608 is displayed at a distant focal plane. For the user, focal plane image 604 may appear to be at a distance f from the user. 1 At a distance f from the user, the focal plane image 606 may appear as 2 and the focal plane image 608 may appear to be at a distance f from the user. 3 Place.

[0141] The processing method used in the example of Figure 3-7 is Figure 8 As shown in the flowchart of , at 802, a number N of texture images are captured at different focal lengths. At 804, a separate depth map is generated for each of the N texture images. At step 806, a focus weight map is generated based on each of the depth maps. At step 808, each texture image is multiplied by the associated focus weight map to form N focal plane images. At step 810, the N focal plane images are rendered on a multi-focal plane display. In the case of a display with motion parallax enabled, the focal plane images may be laterally shifted and / or scaled relative to each other in response to lateral movement of the viewer to simulate motion parallax.

[0142] Example focus weight determination.

[0143] In the example shown in Figure 3-7, for the focal length f i At each of the displayed focal planes, there is a focal plane with focal length f i A single texture image is captured and the focal length f i A single depth map is captured. (Alternatively, other depth maps and texture images are captured but not used to generate the displayed focus plane image.) Under such conditions, the focus weights used to populate the focus weight map can be Figures 13A-13C The method shown in is used to calculate.

[0144] Fig.13A is for a focal length with minimum focal length f 1 Example focus weight w for the focal plane (closest to the camera or observer) 1 In this example, each focus weight w 1 (x,y) is determined only by the fact that it also has focal length f 1 The corresponding pixel depth z captured by the depth camera 1 (x, y). In this example, the second focal plane is at focal length f 2 In this example, the focus weight W1 (x,y) is calculated as follows.

[0145] If z 1 (x,y)≤f 1 , then w 1 (x,y)=1.

[0146] If f 1 ≤z 1 (x,y)≤f 2 , then w 1 (x,y)=[z 1 (x,y)-f 2 ] / [f 1 -f 2 ].

[0147] If z 1 (x,y)≥f 2 , then w 1 (x,y)=0.

[0148] Fig. 13B is for a focal length f i Example focus weight w for the focal plane i (x, y) is a graph where the focal plane is neither the closest nor the farthest focal plane. i (x,y) is the depth z of the corresponding pixel captured only by the depth camera i (x, y), the depth camera also has a focal length f i The closer focal plane is at focal length f i-1 , while the farther focal plane is at focal length f i+1 In this example, the focus weight w i (x,y) is calculated as follows.

[0149] If z i (x,y)≤f i-1 , then w i (x,y)=0.

[0150] If f i-1 ≤z i (x,y)≤f i , then w i (x,y)=[z i (x,y)–f i-1 ] / [f i –f i-1 ].

[0151] If f i ≤z i (x,y)≤f i+1 , then wi (x,y)=[z i (x,y)–f i+1 ] / [f i –f i+1 ].

[0152] If z i (x,y)≥f i+1 , then w i (x,y)=0.

[0153] Fig. 13C is for a focal length f N Example focus weight w for the focal plane N (x, y) is the farthest focal plane. In this example, each focus weight w N (x,y) is the depth z of the corresponding pixel captured only by the depth camera N (x, y), the depth camera also has a focal length f N The closer focal plane is at focal length f i-1 , while the farther focal plane is at focal length f i+1 In this example, the focus weight w N (x,y) is calculated as follows.

[0154] If z N (x,y)≤f N-1 , then w N (x,y)=0.

[0155] If f N-1 ≤z N (x,y)≤f N , then w N (x,y)=[z N (x,y)–f N-1 ] / [f N –f N-1 ].

[0156] If z N (x,y)≥f N , then w N (x,y)=1.

[0157] Figures 13A-13C The above description provides f representing the linear distance from the camera or observer i However, in some embodiments, f i and z i The focus weight w is calculated by taking the inverse of the distance of the value of i (x, y). In these embodiments, the Figures 13A-13CGiven the equation, it can be understood that f i The minimum value (e.g. f 1 ) will represent the farthest focal plane, f i The maximum value (for example, f N ) will denote the closest focal plane. Such embodiments using the opposite distance scale can more easily accommodate using a focal plane located at "infinity".

[0158] Figures 14A-14C Similar to Figures 13A-13C ,but Figures 14A-14C The focusing weight w is shown i (x,y) does not need to be relative to depth z i Piecewise linear. Figures 14A-14C The focus weights are shown relative to the depth z i is piecewise sinusoidal, but other relationships may be achieved in other embodiments. In other example embodiments, Figures 13A-13C In the embodiments of 14A-14C, W i In f i It has a maximum value at f i decreases or remains constant on either side.

[0159] In some embodiments, this may not be the case, i.e., for focal length f i Each display focal plane at i A single texture image is captured and the focal length is f i The captured depth map. For example, at focal length f i There can be a display plane at the same focal length f i No texture image and / or depth map is captured at this location. Similarly, depth maps and texture images can be captured with different focal lengths. Fig. 9 An example of such a condition is shown in , where there are two different depth maps, three different texture images, and two different focal plane images, none of which have the same corresponding focal length.

[0160] Under such conditions, image processing in some embodiments may be performed as follows. The pixel value (eg, brightness value or RGB value) at position (x, y) in the focal plane image i may be obtained by q i (x, y). The pixel values ​​in different display focal planes j can be represented by p j (x,y) represents each pixel value q i (x,y) can be calculated as follows:

[0161]

[0162] where w ij (x,y) is the focus weight in the focus weight map. Weight wij (x,y) can be used to calculate the i The weight w is determined by the depth map represented by (x, y). ij (x,y) represents the captured pixel p from texture image j j (x,y) Display pixel q in focus plane image i i The weight of the contribution of (x,y).

[0163] In some embodiments, the weights are determined based on at least two factors: (i) a factor based on the difference between the focal lengths of focal plane i and captured texture image j, and (ii) a factor based on the focus levels of individual pixels in the captured texture image.

[0164] When both the focal plane i and the texture image j have the same focal length, a factor based on the difference between the focal lengths of the focal plane i and the captured texture image j may have a value of 1, and may decrease as the difference between the focal lengths increases.

[0165] A factor based on the focus level of each pixel in the captured texture image can depend on the difference between the focal length of the texture image and the measured depth of the captured pixel. The factor can have a value of 1 when the measured depth of the captured pixel is equal to the focal length of the texture image, otherwise the factor can be reduced. If a depth map is not captured at the same focal length as the texture image, the measured depth of the captured pixel can be determined, for example, by linear interpolation based on the depth map with the nearest focal length. In some embodiments, as described in more detail below, a defocus map is used to determine the focus level of each pixel. Such embodiments do not require the use of a captured depth map.

[0166] The defocus map is used to form a focus weight map.

[0167] In some embodiments, as described above, in order for the occluded information to eventually reach the focal plane, depth sensing is performed using an aperture of non-negligible size rather than using a pinhole aperture. In some such embodiments, a set of depth maps may be captured using the same aperture and focal length as used to capture the depth image. In an alternative embodiment, filtering of the focal stack image is performed to capture information from the occluded area, which may appear in any focal stack image, and used to form the extended MFP. Such embodiments may be implemented without using a separate depth sensor.

[0168] In some embodiments, a focus weight map is derived for each captured texture image using a "depth from focus" method, such as the method described in Shaojie Zhuo, Terence Sim, "Defocus map estimation from a single image", Pattern Recognition 44 (2011), pp. 1852-1858.

[0169] In some embodiments, N defocus maps are formed, each corresponding to a texture image (e.g., using the method of Zhuo & Sim). Each defocus map covers the depth range of the entire captured view. A depth blending operation can be used to form a corresponding focus weight map. In such an embodiment, the focus weight map is determined based on the focus level rather than based on the measured depth.

[0170] In some cases, the depth blending function is symmetric, producing the same contribution whether the pixel is in front of or behind the focus (focal plane) distance. Defocus maps inherently have this property.

[0171] It can be noted that for the defocused image, the focal length is also known. Therefore, although the scales are different, the origin of both scales is the same. To meet the convention of depth maps, the defocus map can be inverted before depth blending. This makes it basically a focus map, showing the highest values ​​for highest focus. However, this map can still be called a defocus map.

[0172] Fig.10 An example of interpreting a (defocused) focus map as a depth map and decomposing a test image into three MFPs using a linear filter is shown. At 1002, N different texture images are captured at different focal lengths. At 1004, each texture image is filtered to form a corresponding defocus map. The generation of the defocus map can utilize camera parameters such as aperture and focal length. At 1006, N focus weight maps are generated based on the defocus map. At 1008, each texture image is multiplied by the corresponding focus weight map to form a total of N focal plane images. At step 1010, the focal plane images are rendered on a multi-focal plane display.

[0173] The focus weight map generated by using the defocus plane can largely correspond to the focus weight map generated using the depth map, except for the scale, which is not necessarily linear with distance for the defocus map. Although it is believed that this difference does not have a significant effect, in some embodiments it may be desirable to linearize the brightness scale of the defocus map. As described in Zhuo & Sim, when capturing the texture image, the linearization can be performed using knowledge of the camera parameters.

[0174] Filtering and redistribution are used to form multiple focal planes.

[0175] In some embodiments, filtering and redistribution are used to form the focal plane image.

[0176] Filtering and redistribution can reduce deocclusion when generating an MFP that supports viewpoint changes (e.g., generation of motion parallax and / or stereoscopic views). The redistribution operation is used to separate and redistribute the high-frequency and low-frequency components of each focal plane image by filtering. The high frequencies are kept at the same level / distance at which they appear, but the low-frequency components are distributed in the focal plane image. The redistribution of the low-frequency components is feasible because they make only a small contribution to the depth cues in the human visual system.

[0177] In some embodiments, a stack of texture images is captured at different focal lengths, and the depth position for high frequencies is implied by the known focal length. Information from occluded areas is captured to the MFP, gaining the benefits of redistribution, and no depth map or depth blending is used. In some embodiments, a large aperture image is used in order to capture information from occluded areas. The aperture diameter can be on the order of several centimeters. Filtering and redistribution can be implemented in such a way that the information ends up at the redistributed MFP; the filtering is the same over the entire image area, so it is impossible to exclude information captured from occluded areas. The result does not seem to suffer from the fact that occluded areas near the edges can be seen through the occluding texture, changing the brightness of the corresponding pixels.

[0178] There may be practical limits on the optimal aperture size related to information overlap around edges. In addition to limiting the aperture size as a solution, an image processing based solution can be implemented to display occluded information only when the occluded information is revealed from behind the edge, such as when the focal planes are moved relative to each other for the virtual viewpoints (the amount of movement determines which pixels are revealed or covered).

[0179] Fig.11An example of such a method is shown, at 1102, a large aperture light field is captured from a real or synthetic scene. At 1104, N focal stack images are generated from the light field image, each focal stack image having a different focal length. At 1106, an extended focus image (e.g., a "pinhole" type image) is generated from the light field image. At 1108, a high pass filter is applied to the focal stack image to obtain high frequency image components. At 1110, a low pass filter is applied to the extended focus image to obtain low frequency image components. At 1112, the low frequency image components from the extended focus image are added to the filtered (high frequency) focal stack image, possibly with a scaling factor such as 1 / N. At 1114, a redistributed focal plane image, now including high frequency components (from the original focal stack image) and low frequency components (from the extended focus image), is rendered. In some embodiments, the stack of focal plane images may be shifted relative to each other to produce a stereoscopic view of the scene and / or simulate motion parallax. The distribution of low frequency components in different focal plane images allows for a large amount of shifting before any gaps or voids in the image become visible.

[0180] Fig.12 Another technique for generating a focal plane image from a light field using filtering is shown. At 1202, a large aperture light field is captured from a real or synthetic scene. At 1204, N focal stack images pic1...picN are generated from the light field image, each focal stack image having a different focal length. At 1206, an extended focus image (e.g., a "pinhole" type image) pic.ext is generated from the light field image. (In some embodiments, the extended focus image may be formed by separate focus captures with different focal lengths). At 1208, a low-pass filter is applied to the focal plane image to obtain low-frequency image components pic1.lf...picN.lf. At 1210, a low-pass filter is applied to the extended focus image to obtain a low-frequency image component pic_ext.lf. At 1212, the low-frequency image component from the extended focus image is added to the focal plane image, possibly using a scaling factor such as 1 / N. The resulting images now include their original low-frequency components and the low-frequency contribution from pic_ext.lf; therefore, at 1213, the low-pass filtered images pic1.lf ... picN.lf are subtracted to generate redistributed focal plane images pic1.rd ... picN.rd. At 1214, these redistributed focal plane images are displayed to the user. In some embodiments, the stack of focal plane images can be shifted relative to each other to produce a stereoscopic view of the scene and / or simulate motion parallax. The distribution of low-frequency components in different focal plane images allows for a large amount of shifting before any gaps or voids in the images become visible.

[0181] In some such embodiments, low pass filtering is performed using Gaussian filtering. Fig.12In the example of , after low pass filtering of the focal plane image (at 1208), subtracting the low pass filtered image from the original image (at 1213) has the effect of high pass filtering. In an alternative embodiment, high pass filtering of the focal plane image is performed explicitly.

[0182] Signaling multiple depth and focus images.

[0183] Embodiments described herein may use multiple depth maps and focus images corresponding to a single moment in time. In some embodiments, techniques are used for efficient storage and / or communication of depth maps and focus plane images.

[0184] The depth image and the focus image are associated.

[0185] In some cases, the focal length of the depth capture may be different than the focal length of the image capture. The resolution of the depth map may be different, typically having a lower resolution than the image capture. In some embodiments, during upsampling of the depth map, an edge map in the image map may be used to provide information to refine the depth edges. The depth map may be signaled at a different frame rate and interpolated to the image frame rate. The depth map may also have a different bit depth and mapping of image values ​​to depth values.

[0186] In many cases, the depth map may have little detail except around the edges of objects. In some embodiments, the resolution of the depth map may be reduced for communication, and then resized to full resolution before being used to calculate the depth weighting function. When upsampling the depth map for a particular focal depth value, the presence of a high-resolution image capture may be used to guide the interpolation around the edges. In many cases, the depth map is a single channel image, without color, and the bit depth may be relatively low. The relationship between bit depth and actual distance may be represented by a transfer function.

[0187] Video sequence level parameters.

[0188] Given the possible differences between focal plane images and depth maps, such as bit depth, spatial resolution, temporal frame rate and focal length value, an encoded video sequence comprising multiple focused images and depth maps can provide these parameter values ​​independently for both focused images and depth maps. A description of the sequence level parameters is shown in Table 1.

[0189] Table 1. Multifocus image sequence information parameters.

[0190]

[0191] The focused image sequence level parameters are constant over the sequence and describe properties that are common to all focused images of the time series.

[0192] num_focal_images: The number of focused images corresponding to a single frame time.

[0193] focal_images_fps: The frame rate of the focused images sequence defines the time offset between images corresponding to the same focal length.

[0194] focal_images_height: The spatial height of each focused image in pixels.

[0195] focal_images_width: The spatial width of each focused image in pixels.

[0196] focal_images bit_depth: The bit depth of samples for each focal image.

[0197] focal_images_num_color_planes: describes the number of color planes, e.g. 3 for RGB or YUV sequences.

[0198] focal_distance[f]: Each entry of this array gives the focal distance corresponding to the index of the focal image.

[0199] The depth map sequence-level parameters are constant over the sequence and describe characteristics common to the depth maps of the sequence.

[0200] num_depth_maps: Number of depth maps corresponding to a single frame time. May be different from the number of focus images.

[0201] depth_maps_fps: The frame rate of the depth map sequence defines the time offset between images corresponding to the same depth distance.

[0202] depth_maps_height: The spatial height of each depth map in pixels.

[0203] depth_maps_width: The spatial width of each depth map in pixels.

[0204] depth_maps_bit_depth: The bit depth of samples for each depth map image.

[0205] depth_map_distance[f]: Each entry of this array gives the distance value corresponding to an index into the depth map. This is the distance corresponding to the focal length used when recording the depth map.

[0206] Frame level parameters.

[0207] Individual frames in a video sequence may indicate their type: focused image or depth map, index the relevant sequence level parameter set, and additionally indicate the temporal offset via the image count, and an index to the focal_distance or depth_map_distance value. These frame level parameters are shown in Tables 2 and 3.

[0208] Table 2. Aggregate image single frame parameters.

[0209]

[0210] The frame-level parameters for a single focused image are described below:

[0211] sequence_id: refers to a single sequence parameter set used in case of multiple sequence parameter sets.

[0212] frame_count: describes the time position of the focused image within the frame sequence.

[0213] depth_map_distance_index: A sequence-level list of indexed focal length values.

[0214] Table 3. Depth map single frame parameters.

[0215]

[0216] The frame-level parameters for a single depth map are described below:

[0217] sequence_id: refers to a single sequence parameter set used in case of multiple sequence parameter sets.

[0218] frame_count: describes the time position of the focused image within the frame sequence.

[0219] depth_map_distance_index: A sequence-level list of indexed depth map distance values.

[0220] Inter-image prediction is used in encoding the focus plane image.

[0221] Correlation between images captured under different focus conditions can be developed via inter-image prediction using techniques similar to SNR scalability, where quality varies but resolution does not. In some embodiments, correlation between different focus captures of the same scene is developed by signaling one focus capture image and signaling the difference between the first focus capture image and a second focus capture image.

[0222] Inter-frame depth map prediction is used in encoding.

[0223] The correlation between depth maps can be used to reduce bandwidth requirements.Similar to signaling a single base focused image and additional focused images via the residual, multiple depth maps with different focus captures can be signaled by predicting between the depth maps.

[0224] Additional features in some embodiments.

[0225] In some embodiments, the number and position of the formed focal planes are the same as the number and position of the captured texture images. In the case of different numbers and / or positions, the texture images can first be blended to the nearest focal plane according to their distance from the corresponding focal plane position.

[0226] It is worth noting that in various MFP methods, a depth map is used to separate or decompose scene information / pixels into multiple depth ranges for forming corresponding focal planes. Instead of a depth map, other depth-dependent mapping criteria can be used to perform the separation. An example of an optional depth-dependent mapping is described above with respect to the use of a defocus map for this purpose. A defocus map is similar to a depth map, but instead of depth sensing, the defocus map is based on image blur, which can be detected by filtering the image.

[0227] Another criterion for separation in some embodiments is to use depth of field. However, depth of field follows relatively complex 3D and optical geometry mathematics. DoF appears in the image as a region (hyperplane) with focused pixels, while the outer regions are correspondingly defocused. By using appropriate filtering to detect the focused regions, calculating DoF can be replaced by detecting the focused regions by filtering.

[0228] In an embodiment where redistribution of spatial frequency components is performed, a stack of texture images is captured with different focal lengths, and the depth position of the high frequencies is implied by the known focal length, which is now used as a criterion for assigning depth information. Furthermore, filtering is used to detect a set of complementary DoFs and corresponding focal stack images, covering the entire captured volume in depth and focus information. The number and positions of the focal images can be mathematically determined so as to capture most of the in-focus details (high frequencies) of the scene.

[0229] In some embodiments, a method includes obtaining a plurality of texture images of a scene, each texture image having a different corresponding focal length; and for each texture image, generating a focal plane by: (i) determining a corresponding weight for each of a plurality of pixels of the texture image, wherein the weight represents an amount that the pixel is in focus, and (ii) multiplying a pixel value of each of the plurality of pixels by the corresponding weight. The focal plane images may be displayed on a multiple focal plane display, for example, substantially simultaneously or in a time-multiplexed manner (e.g., sequentially).

[0230] In some embodiments, a method includes obtaining a plurality of texture images p of a scene. i , each texture image has a different corresponding focal length d i ; and for each texture image p i , the focal plane image q is generated by i :(i) Determine the corresponding weight w of each pixel in the texture image i , where the weight w i (x,y) represents the amount of focus on pixel (x,y), and (ii) the texture image p i Each pixel value p i (x,y) multiplied by the corresponding weight w i (x,y) to generate the focal plane image q i , so that q i (x,y)=p i (x,y)·w i (x,y).

[0231] The depth value z corresponding to the pixel may be based at least in part on i (x, y) and the focal length d of the texture image that includes the pixel i The difference between determines the amount of pixel focus in the texture image.

[0232] In some embodiments, for each texture image, a depth image z of the scene is obtained. i (x,y). For each texture image p i (x,y), through the function w i [z i (x,y)] to determine the weight w i (x, y). In some embodiments, a single depth image may be obtained for use with all texture images, and z i (x, y) may be the same for all values ​​of i. In some embodiments, w i [z i (x,y)] at w i [d i ] has the maximum value.

[0233] In some embodiments, obtaining a plurality of texture images comprises: receiving a set of initial texture images at a display device having a plurality of display focal planes, each display focal plane having a different corresponding focal length; and selecting, from the set of initial texture images, a selected texture image p having a focal length corresponding to the focal length of the display focal plane (e.g., having the same focal length or a closest focal length). i Each selected texture image p i may have a focal length d equal to one of the display focal planes i .

[0234] In some embodiments, a method of providing a multi-layer image of a scene includes: for each of a plurality of different focal lengths, (i) capturing a texture image of the scene focused at the corresponding focal length, and (ii) capturing a depth image of the scene focused at the corresponding focal length (e.g., using a time-of-flight camera); and transmitting the captured texture image and depth image. Each texture image and the corresponding corresponding depth image may be captured substantially simultaneously. Each texture image and the corresponding corresponding depth image are captured using the same optical device. In some embodiments, the captured texture image and depth image are encoded in a bitstream, and transmitting the captured texture image and depth map includes transmitting the encoded bitstream. In some such embodiments, encoding the captured texture image and depth image includes using at least a first texture image of the texture images as a predictor for encoding at least a second texture image of the texture images. In some embodiments, encoding the captured texture image and depth image includes using at least one texture image of the texture images as a predictor for encoding at least one depth map of the depth image.

[0235] Note that the various hardware elements of one or more described embodiments are referred to as "modules" that perform (i.e., execute, implement, etc.) the various functions described herein in conjunction with the corresponding modules. As used herein, a module includes hardware that a person skilled in the relevant art deems suitable for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices). Each described module may also include instructions executable to perform one or more functions described as being performed by the corresponding module, and note that these instructions may take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, and / or software instructions, etc., and may be stored in any suitable non-transitory computer-readable medium or media, such as what is commonly referred to as RAM, ROM, etc.

[0236] Although the features and elements are described above in specific combinations, it will be understood by those skilled in the art that each feature or element can be used alone or in any combination with other features and elements. In addition, the methods described herein can be implemented in a computer program, software, or firmware that is incorporated into a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). A processor associated with the software can be used to implement a radio frequency transceiver used in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method comprising: obtaining a plurality of texture images of a scene, each texture image having a different corresponding focal length; filtering each of the plurality of texture images to form a corresponding defocus map for each texture image; for each texture image of the plurality of texture images and for each corresponding defocus map, generating a corresponding focus weight map from the defocus map, wherein each corresponding focus weight map includes a plurality of focus weights, the plurality of focus weights representing an amount by which each corresponding pixel of a plurality of pixels of the corresponding texture image is in focus, the focus weights of the pixels in the texture image being determined based on the corresponding defocus map; and for each texture image in the plurality of texture images, generating a corresponding focus plane image by multiplying the texture image by the focus weight map, wherein multiplying the texture image by the focus weight map includes multiplying a pixel value of each of the plurality of pixels of the texture image by the corresponding focus weight of the plurality of focus weights in the focus weight map. 2 . The method of claim 1 , further comprising displaying the focal plane images at the respective focal lengths of the focal plane images in a multiple focal plane display. The method of claim 2 , wherein the focal plane images are displayed substantially simultaneously.

4. The method according to any one of claims 2 or 3, further comprising: For each of the focal plane images, a respective texture image is selected from the plurality of texture images.

5. The method according to claim 1, wherein obtaining a plurality of texture images comprises: Each of the plurality of texture images is captured at the respective focal length. The method of claim 1 , wherein generating each of the corresponding focus weight maps is further based on a focus level of the respective texture image.

7. The method of any one of claims 1-6, further comprising generating a virtual viewpoint by shifting at least one of the focal plane images by an amount inversely proportional to a display focal length of the corresponding focal plane image.

8. The method of claim 7, further comprising displaying the generated virtual viewpoint as one of a stereoscopic viewpoint pair.

9. The method of claim 7, further comprising displaying the generated virtual viewpoint in response to viewer head movement to simulate motion parallax.

10. A system comprising a processor and a computer readable medium storing instructions, which instructions, when executed, are operable to cause the system to perform the method according to any one of claims 1-9.

Images