Providing segmentation information for immersive video

Abstract

Systems and methods are provided in which a plurality of source views of a scene are encoded as an immersive video comprising a plurality of tiles. The tile is segmented based on segmentation information from at least one of the source views. The segmentation information associates each of a plurality of regions of the source view with a respective entity. Information is also encoded that indicates which of the source views are associated with the segmentation information for segmenting the tile. This information may be used to provide improved rendering, particularly where rendering is performed on less than all entities in the scene.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to European patent application No. EP22305627.6 filed on April 27, 2022, the entire text of which is incorporated herein by reference. Background Art

[0003] Virtual reality has gained more attention with the advent of consumer head-mounted displays (Oculus, HTC Vive, etc.) associated with improved screen resolution, field of view, and frame rate. This phenomenon has opened the door to new applications in many areas, including gaming, education / training, and multimedia.

[0004] In this area, new types of video content are emerging, including 360° videos. Such videos allow users to see everything around themselves by pure rotation around a fixed viewpoint. Even if pure rotation is sufficient for the first omnidirectional video experience, it may quickly become frustrating for viewers who would expect more freedom. Moreover, since head rotation includes small translations of the head that are not reproduced by such experiences, such experiences may also cause vertigo. Figure 3 A camera rig that can be used to capture volumetric video is illustrated.

[0005] An alternative to these 360° content is called volumetric or 6 degrees of freedom (6DoF) video. When watching such a video, in addition to rotating, the user can also pan their head within the viewed content and experience parallax. Such videos significantly increase the sense of immersion and the perception of depth of the scene, but can also reduce vertigo by providing consistent visual feedback during head translation. Associated content can be created with specialized sensors, allowing the color and geometry of the scene of interest to be recorded simultaneously. Using a color camera rig combined with photogrammetry techniques is one way to do this recording ( Figure 3 ).

[0006] While a 360° video may be a temporal sequence of specific images resulting from the unmapping of a spherical texture (eg, lat-lon / equirectangular images), a 6DoF video "frame" is more complex as it should embed information from several viewpoints.

[0007] Depending on the viewing conditions, two different types of volumetric video can be considered. The more permissive volumetric video allows completely free navigation within the video content, while the second volumetric video restricts the user viewing space to a limited volume. The latter context is a natural compromise between free navigation and passive viewing conditions for audience members sitting in their armchairs. Moreover, it is conceivable in a shorter time and it requires less effort to acquire the content (due to the limitations of the visualization box) than the free viewpoint scenario. A specific implementation of stereoscopic video with limited viewing space is provided within MPEG as an extension of V3C, which is called MPEG for Immersive Video (MIV) and belongs to the MPEG-I standard suite. For V3C, see ISO / IEC 23090-5:2022 (2E) Visual Volumetric Video Coding (V3C) and Video-based Point Cloud Compression (V-PCC). For MIV, see ISO / IEC 23090-12:2022 MPEG Immersive Video.

[0008] The current MIV allows volume information to be stored as an atlas frame ( FIG. 4A to FIG. 4B ), which are then "video" encoded using a conventional codec (usually HEVC). Each tile typically represents the projection of a sub-portion of the 3D input scene as a combination of color, geometry, and transparency 2D attributes, and the set of all tiles is designed in the encoding phase to "cover" the entire scene while avoiding redundancy where possible. In the decoding phase, the atlas is "video" decoded and the tiles are rendered in a view synthesis process to recover the viewport associated with the desired viewing position.

[0009] Figure 4A An example of a color atlas generated using MIV is illustrated. Figure 4B The corresponding geometry atlas is illustrated.

[0010] In the first version of MIV, the texture (color) and geometry (depth) components of the encoded volumetric video are spatially aligned. This assumes that the multi-view plus depth (MVD) content (which consists of multiple views captured by real or virtual cameras with associated depth maps) input to the MIV encoder is such that the texture (color) image and depth map of a given view share the same camera parameters.

[0011] At the MPEG 137 meeting (January 2022), it was proposed that the second edition of MIV in preparation will support content where the texture and depth components are not necessarily aligned (for example, where they are captured by separate input devices). See Jill Boyce, Basel Salahieh "New work areas for MIV v2", ISO / IEC JTC 1 / SC 29 / WG 4m58560, January 2022. See also "Use cases and requirements for MIV. Edition-2", ISO / IEC JTC 1 / SC 29 / WG 2N00157, January 2022. Summary of the invention

[0012] Embodiments described herein include methods for video encoding and decoding (collectively referred to as "decoding"). In methods according to some embodiments, multiple source views of a scene are obtained. For at least one source view in the source views, segmentation information is obtained that associates each of multiple regions of the source view with a corresponding entity. The multiple source views are encoded into an immersive video including multiple blocks, and the blocks are segmented according to the segmentation information. Information indicating which source views in the source views are associated with the segmentation information for segmenting the blocks is also encoded.

[0013] The embodiments described herein also include an apparatus comprising one or more processors configured to perform the following method. A plurality of source views of a scene are obtained. For at least one source view in the source views, segmentation information is obtained that associates each of a plurality of regions of the source view with a corresponding entity. The plurality of source views are encoded into an immersive video comprising a plurality of tiles, the tiles being segmented according to the segmentation information. Information indicating which of the source views in the source views are associated with the segmentation information for segmenting the tiles is also encoded.

[0014] In some embodiments, for each source view, the information indicating which of the source views are associated with the segmentation information used to segment the tile includes a flag indicating whether the input view is associated with the segmentation information used to segment the tile.

[0015] In some embodiments, the segmentation information associated with the source view includes a physical graph associated with the source view.

[0016] In some embodiments, the information indicating which of the source views are associated with the segmentation information used to segment the block includes information indicating whether the segmentation information is based on a depth image or a texture image.

[0017] In a method according to some embodiments, an encoded immersive video is obtained, the encoded immersive video including a plurality of tiles, the video representing a plurality of input views of a scene. Information indicating which of the input views are associated with segmentation information for segmenting the tiles is obtained. The immersive video is rendered according to the information indicating which of the input views are associated with the segmentation information for segmenting the tiles.

[0018] In an apparatus according to some embodiments, the apparatus includes one or more processors configured to perform the following method. Obtain an encoded immersive video, the encoded immersive video including a plurality of tiles, the video representing a plurality of input views of a scene. Obtain information indicating which of the input views are associated with segmentation information for segmenting the tiles. Render the immersive video according to the information indicating which of the input views are associated with the segmentation information for segmenting the tiles.

[0019] In some embodiments, for each source view, the information indicating which of the source views are associated with the segmentation information used to segment the block includes a flag indicating whether the input view is associated with the segmentation information used to segment the block. In some embodiments, the information indicating which of the source views are associated with the segmentation information used to segment the block includes information indicating whether the segmentation information is based on a depth image or a texture image.

[0020] In some embodiments, rendering the immersive video based on the information indicating which of the input views are associated with the segmentation information used to segment the block includes: identifying at least one selected entity to be rendered; and in response to determining that the segmentation information is based on a depth image, performing warping of depth pixels of the immersive video only for the depth block associated with the at least one selected entity.

[0021] In some embodiments, rendering the immersive video based on the information indicating which of the input views are associated with the segmentation information used to segment the block includes: identifying at least one selected entity to be rendered; and in response to determining that the segmentation information is based on the texture image: performing warping of depth pixels of the immersive video for a depth block that includes at least a depth block associated with the at least one selected entity; and performing blending of color values ​​based only on color pixels associated with the at least one selected entity.

[0022] In some embodiments, rendering the immersive video based on the information indicating which of the input views are associated with the segmentation information used to segment the block includes: in response to determining that the segmentation information is based on the depth image, making a visibility determination based at least in part on the segmentation information.

[0023] In some embodiments, rendering the immersive video based on information indicating which of the input views are associated with segmentation information used to segment the tile includes: in response to determining that the segmentation information is based on a texture image, making a visibility determination at least in part based on the segmentation information during a blending stage.

[0024] The signal according to some embodiments includes: an encoded immersive video, the encoded immersive video including a plurality of tiles, the video representing a plurality of input views of a scene; and information indicating which of the input views are associated with the segmentation information for segmenting the tile. In some such embodiments, for each source view, the information indicating which of the source views are associated with the segmentation information for segmenting the tile includes a flag indicating whether the input view is associated with the segmentation information for segmenting the tile.

[0025] Some embodiments include a computer readable medium storing a signal as described herein.

[0026] Some embodiments include a computer readable medium comprising instructions for causing one or more processors to perform a method as described herein.

[0027] Some embodiments include a computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform a method as described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1A is a system diagram illustrating an example communication system in which one or more disclosed embodiments may be implemented.

[0029] Figure 1B is an example of a method that can be used according to an embodiment of the present invention. Figure 1A A system diagram of an example wireless transmit / receive unit (WTRU) for use within the illustrated communication system.

[0030] Figure 1C is a functional block diagram of a system used in some embodiments described herein.

[0031] Figure 2A is a functional block diagram of a block-based video encoder, such as the encoder used for VVC.

[0032] Figure 2B is a functional block diagram of a block-based video decoder, such as the decoder used for VVC.

[0033] Figure 3 A camera rig that can be used to capture volumetric video is illustrated.

[0034] Figure 4A An example of a color atlas generated using MIV is illustrated. Figure 4B The corresponding geometry atlas is illustrated.

[0035] Figure 5A An example of a texture map for a scene is illustrated. Figure 5B An example of a depth map of a scene is illustrated. Figure 5C An example of an entity graph for a scene is illustrated.

[0036] Fig. 6A An example of a texture tile atlas after TMIV encoding of an MVD sequence is illustrated. Figure 6B The corresponding depth block atlas is illustrated.

[0037] Fig. 7A An example of a texture tile atlas after TMIV encoding of selected entities of an MVD sequence is illustrated. Figure 7B The corresponding depth block atlas is illustrated.

[0038] Figure 8 The different coordinates generated when the color camera and the depth camera have different positions are schematically illustrated.

[0039] FIG. 9A to FIG. 9B An example of a camera device is illustrated in which the color sensor and the depth sensor have different locations. Fig. 9A A smart phone is illustrated. Fig. 9B A depth camera is illustrated.

[0040] Fig.10 Schematically illustrates a camera array layout that may be used to capture a view of a scene in accordance with some embodiments.

[0041] Fig.11 An example of a complete MIV processing chain from 3D scene acquisition to synthesis of a virtual viewport is illustrated.

[0042] Fig.12 is a schematic diagram of the reprojection process.

[0043] Fig.13 is a flowchart of an encoding method performed according to some embodiments.

[0044] Fig.14 is a flowchart of a decoding method performed according to one embodiment.

[0045] Fig.15 is a schematic diagram of the relationship between different data in some implementation schemes.

[0046] Fig.16 is a flow chart of a decoding method performed according to some embodiments. DETAILED DESCRIPTION

[0047] Figure 1A 1 is a diagram illustrating an example communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content such as voice, data, video, messaging, broadcast, etc. to multiple wireless users. The communication system 100 may enable multiple wireless users to access such content through sharing of system resources (including wireless bandwidth). For example, the communication system 100 may employ one or more channel access methods such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single carrier FDMA (SC-FDMA), zero tail unique word DFT spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block filtered OFDM, filter bank multi-carrier (FBMC), etc.

[0048] like Figure 1AAs shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104, CN 106, public switched telephone network (PSTN) 108, Internet 110 and other networks 112, but it should be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d (any of which may be referred to as a “station” and / or “STA”) may be configured to send and / or receive wireless signals and may include user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular phone, a personal digital assistant (PDA), a smart phone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (IoT) device, a watch or other wearable device, a head-mounted display (HMD), a vehicle, a drone, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in an industrial and / or automated process chain environment), a consumer electronic device, a device operating on a commercial and / or industrial wireless network, etc. Any of the WTRUs 102a, 102b, 102c, and 102d may be interchangeably referred to as a UE.

[0049] The communication system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN 106, the Internet 110, and / or other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node B, an eNode B, a Home Node B, a Home eNode B, a gNB, an NR Node B, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each depicted as a single element, it will be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0050] The base station 114a may be part of the RAN 104, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), a relay node, etc. The base station 114a and / or the base station 114b may be configured to send and / or receive wireless signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be in a licensed spectrum, an unlicensed spectrum, or a combination of licensed and unlicensed spectrums. A cell may provide coverage of wireless services to a specific geographic area, which may be relatively fixed or may change over time. The cell may be further divided into cell sectors. For example, a cell associated with the base station 114a may be divided into three sectors. Therefore, in one embodiment, the base station 114a may include three transceivers, i.e., one transceiver for each sector of the cell. In an embodiment, the base station 114a may employ multiple-input multiple-output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to send and / or receive signals in a desired spatial direction.

[0051] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0052] More specifically, as noted above, the communication system 100 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, the base station 114a in the RAN 104 and the WTRUs 102a, 102b, 102c may implement a radio technology, such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), that may use Wideband CDMA (WCDMA) to establish the air interface 116. WCDMA may include communication protocols, such as High Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High Speed ​​UL Packet Access (HSUPA).

[0053] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA) that may establish the air interface 116 using Long Term Evolution (LTE) and / or Advanced LTE (LTE-A) and / or Advanced LTE Pro (LTE-APro).

[0054] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as NR radio access, which may establish the air interface 116 using New Radio (NR).

[0055] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may together implement LTE radio access and NR radio access, for example using the dual connectivity (DC) principle. Thus, the air interface used by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions transmitted to / from multiple types of base stations (e.g., eNBs and gNBs).

[0056] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi)), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), etc.

[0057] Figure 1AThe base station 114b in the may be, for example, a wireless router, a Home NodeB, a Home eNodeB, or an access point, and may utilize any suitable RAT to facilitate wireless connectivity in a local area, such as a business location, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a road, and the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology (such as IEEE 802.11) to establish a wireless local area network (WLAN). In an embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology (such as IEEE 802.15) to establish a wireless personal area network (WPAN). In another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish a microcell or a femtocell. As Figure 1A As shown, the base station 114b may have a direct connection to the Internet 110. Therefore, the base station 114b may not be required to access the Internet 110 via the CN 106.

[0058] The RAN 104 may be in communication with the CN 106, which may be any type of network configured to provide voice, data, applications, and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have different quality of service (QoS) requirements, such as different throughput requirements, delay requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. The CN 106 may provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, etc., and / or perform high-level security functions, such as user authentication. Although not described in detail in the accompanying drawings, the CN 106 may be configured to provide voice, data, applications, and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. Figure 1A Although not shown in the figure, it will be appreciated that the RAN 104 and / or the CN 106 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 104 or a different RAT. For example, in addition to being connected to the RAN 104, which may utilize NR radio technology, the CN 106 may also be in communication with another RAN (not shown) that employs GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0059] The CN 106 may also act as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network that provides plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the Transmission Control Protocol (TCP), the User Datagram Protocol (UDP), and / or the Internet Protocol (IP) in the TCP / IP Internet protocol suite. The networks 112 may include wired communication networks and / or wireless communication networks owned and / or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs, which may employ the same RAT as the RAN 104 or a different RAT.

[0060] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communication system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). Figure 1A The illustrated WTRU 102c may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[0061] Figure 1B is a system diagram illustrating an example WTRU 102. Figure 1B As shown, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138. It will be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0062] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal decoding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. Although Figure 1B The processor 118 and the transceiver 120 are depicted as separate components, but it is understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0063] The send / receive element 122 may be configured to send a signal to a base station (e.g., base station 114a) or receive a signal from the base station via an air interface 116. For example, in one embodiment, the send / receive element 122 may be an antenna configured to send and / or receive an RF signal. In an embodiment, the send / receive element 122 may be a transmitter / detector configured to send and / or receive, for example, IR, UV or visible light signals. In another embodiment, the send / receive element 122 may be configured to send and / or receive both RF signals and optical signals. It should be understood that the send / receive element 122 may be configured to send and / or receive any combination of wireless signals.

[0064] Although the transmit / receive element 122 Figure 1B 1 as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0065] The transceiver 120 may be configured to modulate signals to be transmitted by the transmit / receive element 122 and to demodulate signals received by the transmit / receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. For example, the transceiver 120 may include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0066] The processor 118 of the WTRU 102 may be coupled to and may receive user input data from a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from and store data in any type of suitable memory, such as a non-removable memory 130 and / or a removable memory 132. The non-removable memory 130 may include a random access memory (RAM), a read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from and store data in a memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[0067] The processor 118 may receive power from the power source 134, and may be configured to distribute and / or control power to other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0068] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to or in lieu of the information from the GPS chipset 136, the WTRU 102 may receive location information from a base station (e.g., base stations 114a, 114b) over the air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information by any suitable location-determination method while remaining consistent with an embodiment.

[0069] The processor 118 may also be coupled to other peripherals 138, which may include one or more software modules and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, module, a frequency modulation (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, etc. Peripheral device 138 may include one or more sensors, which may be one or more of the following: a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0070] The WTRU 102 may include a full-duplex radio for which transmission and reception of some or all signals (e.g., associated with specific subframes for both UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous. The full-duplex radio may include an interference management unit for reducing and / or substantially eliminating self-interference via signal processing performed via hardware (e.g., a choke) or via a processor (e.g., a separate processor (not shown) or via the processor 118). In an embodiment, the WRTU 102 may include a half-duplex radio for which transmission and reception of some or all signals (e.g., associated with specific subframes for both UL (e.g., for transmission) or downlink (e.g., for reception)) may be concurrent and / or simultaneous.

[0071] Although the WTRU Figure 1A to Figure 1B Although described as wireless terminals, it is contemplated that in certain representative embodiments, such terminals may (eg, temporarily or permanently) use a wired communications interface with a communications network.

[0072] In a representative embodiment, the other network 112 may be a WLAN.

[0073] Given that Figure 1A to Figure 1B As well as the corresponding description, one or more or all of the functions described herein may be performed by one or more simulation devices (not shown). A simulation device may be one or more devices configured to emulate one or more or all of the functions described herein. For example, a simulation device may be used to test other devices and / or simulate network and / or WTRU functions.

[0074] The simulation device may be designed to implement one or more tests of other devices in a laboratory environment and / or in an operator network environment. For example, one or more simulation devices may perform one or more functions or all functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network in order to test other devices within the communication network. One or more simulation devices may perform one or more functions or all functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. The simulation device may be directly coupled to another device for testing purposes and / or may use over-the-air wireless communications to perform testing.

[0075] One or more simulation devices can perform one or more (including all) functions without being implemented / deployed as part of a wired and / or wireless communication network. For example, the simulation device can be used in a test scenario in a test laboratory and / or a non-deployed (e.g., testing) wired and / or wireless communication network to implement testing of one or more components. One or more simulation devices can be test equipment. Direct RF coupling and / or wireless communication via RF circuitry (e.g., which can include one or more antennas) can be used by the simulation device to send and / or receive data.

[0076] Example System

[0077] The embodiments described herein are not limited to implementation on a WTRU. Such embodiments may use other systems (such as Figure 1C system) to achieve this. Figure 1C It is a block diagram of an example of a system that implements various aspects and embodiments. System 1000 may be embodied as a device including various components described below, and is configured to perform one or more aspects of the aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances and servers. The elements of system 1000 may be embodied in a single integrated circuit (IC), multiple ICs and / or discrete components individually or in combination. For example, in at least one embodiment, the processing element and encoder / decoder element of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is coupled to one or more other systems or other electronic devices via, for example, a communication bus or by a dedicated input and / or output port communication. In various embodiments, system 1000 is configured to implement one or more aspects of the aspects described in this document.

[0078] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing, for example, various aspects described in this document. The processor 1010 may include embedded memory, input and output interfaces, and various other circuit systems as known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include a non-volatile memory and / or a volatile memory, including but not limited to an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a programmable read-only memory (PROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a magnetic disk drive, and / or an optical disk drive. As non-limiting examples, the storage device 1040 may include an internal storage device, an attached storage device (including a removable and non-removable storage device), and / or a network accessible storage device.

[0079] The system 1000 includes an encoder / decoder module 1030, which is configured to process data, for example, to provide encoded video or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents one or more modules that may be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both of an encoding module and a decoding module. Additionally, the encoder / decoder module 1030 may be implemented as an independent element of the system 1000, or may be combined in the processor 1010 as a combination of hardware and software known to those skilled in the art.

[0080] Program code to be loaded onto the processor 1010 or the encoder / decoder 1030 to perform various aspects described in this document may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of the various items during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded video or partially decoded video, bitstreams, matrices, variables, and intermediate or final results of processing equations, formulas, operations, and operation logic.

[0081] In some embodiments, memory internal to the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store, for example, the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations, such as MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by the Joint Video Experts Team (JVET)).

[0082] Input to the elements of system 1000 may be provided through various input devices as indicated in block 1130. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 1C Other examples not shown include composite video.

[0083] In various embodiments, the input device of block 1130 has associated corresponding input processing elements as known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select a signal band that may be referred to as a channel in certain embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired packet stream. The RF section of various embodiments includes one or more elements for performing these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or to baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal that sends by wired (for example, cable) medium, and filter to required frequency band by filtering, down-conversion and again to perform frequency selection.Each embodiment rearranges the order of (and other) element described above, removes some elements in these elements, and / or adds other elements that perform similar or different functions.Adding element can be included in and inserts element between existing element, such as for example, inserts amplifier and analog-to-digital converter.In various embodiments, the RF part comprises antenna.

[0084] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices across the USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, in a separate input processing IC or in the processor 1010, if necessary. Similarly, USB or HDMI interface processing aspects may be implemented, if necessary, in a separate interface IC or in the processor 1010. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 1010 and the encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data stream as needed for presentation on an output device.

[0085] The various components of system 1000 may be disposed within an integrated housing. Within the integrated housing, the various components may be interconnected and transmit data between the components using a suitable connection arrangement 1140 (e.g., an internal bus known in the art, including an inter-chip (I2C) bus, wiring, and a printed circuit board).

[0086] The system 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data through the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0087] In various embodiments, data is streamed or otherwise provided to the system 1000 using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signals of these embodiments are received by a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 of these embodiments is typically connected to an access point or router that provides access to an external network (including the Internet) for allowing streaming applications and other over-the-top communications. Other embodiments provide streaming data to the system 1000 using a set-top box that delivers data via an HDMI connection of an input box 1130. Still other embodiments provide streaming data to the system 1000 using an RF connection of an input box 1130. As described above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0088] System 1000 can provide output signals to various output devices, including display 1100, speaker 1110 and other peripheral devices 1120. Display 1100 of various embodiments includes one or more of, for example, a touch screen display, an organic light emitting diode (OLED) display, a curved display and / or a foldable display. Display 1100 can be used for a television, a tablet computer, a laptop, a cellular phone (mobile phone) or other devices. Display 1100 can also be integrated with other components (e.g., as in a smart phone), or it can be independent (e.g., an external display for a laptop). In various examples of embodiments, other peripheral devices 1120 include one or more of an independent digital video disc (or digital versatile disc) (DVR, a term for both), a disc player, a stereo system and / or a lighting system. Various embodiments use one or more peripheral devices 1120, which provide functions based on the output of system 1000. For example, a disc player performs the function of playing the output of system 1000.

[0089] In various embodiments, control signals are transmitted between the system 1000 and the display 1100, speaker 1110, or other peripheral device 1120 using signaling such as AV.Link, consumer electronics control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices can be communicatively coupled to the system 1000 via dedicated connections through corresponding interfaces 1070, 1080, and 1090. Alternatively, the output devices can be connected to the system 1000 via the communication interface 1050 using the communication channel 1060. The display 1100 and the speaker 1110 can be integrated into a single unit with other components of the system 1000 in an electronic device (such as, for example, a television). In various embodiments, the display interface 1070 includes a display driver, such as, for example, a timing controller (T Con) chip.

[0090] For example, if the RF portion of input 1130 is part of a stand-alone set-top box, the display 1100 and speaker 1110 may alternatively be independent of one or more of the other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0091] These embodiments may be performed by computer software implemented by the processor 1010 or by hardware or by a combination of hardware and software. As a non-limiting example, these embodiments may be implemented by one or more integrated circuits. As a non-limiting example, the memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology (such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory). As a non-limiting example, the processor 1010 may be of any type suitable for the technical environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0092] Detailed Description

[0093] Block-based video decoding

[0094] Similar to HEVC, VVC is built on a block-based hybrid video coding framework. Figure 2A A block diagram of a block-based hybrid video coding system 200 is presented. Variations of the encoder 200 are contemplated, but for clarity, the encoder 200 is described below without describing all contemplated variations.

[0095] Before being encoded, the video sequence may undergo pre-encoding processing (204), such as applying a color transform to an input color picture (e.g., a transform from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of input picture components to make the signal distribution more resilient to compression (e.g., using histogram equalization of one color component). Metadata may be associated with the pre-processing and attached to the bitstream.

[0096] The input video signal 202 including the picture to be encoded is partitioned (206) and processed block by block in units such as CUs. Different CUs may have different sizes. In VTM-1.0, a CU may be up to 128×128 pixels. However, unlike HEVC, which partitions blocks based only on a quadtree, in VTM-1.0, a coding tree unit (CTU) is partitioned into CUs to accommodate local characteristics of changes based on quad / binary / ternary trees. In addition, the concept of multiple partition unit types in HEVC is removed, so that the separation of CU, prediction unit (PU), and transform unit (TU) no longer exists in VVC-1.0; instead, each CU is always used as a basic unit for both prediction and transformation without further partitioning. In a multi-type tree structure, a CTU is first partitioned by a quadtree structure. Then, each quadtree leaf node may be further partitioned by a binary tree structure and a ternary tree structure. Different partition types may be used, such as quad partitioning, vertical binary partitioning, horizontal binary partitioning, vertical ternary partitioning, and horizontal ternary partitioning.

[0097] exist Figure 2A In an encoder of a video processor, spatial prediction (208) and / or temporal prediction (210) may be performed. Spatial prediction (or "intra-frame prediction") uses pixels from samples of neighboring blocks that have been decoded in the same video picture / slice (which are called reference samples) to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in video signals. Temporal prediction (also known as "inter-frame prediction" or "motion compensated prediction") uses reconstructed pixels from decoded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in video signals. The temporal prediction signal for a given CU may be signaled by one or more motion vectors (MVs) that indicate the amount and direction of motion between the current CU and its temporal reference. Furthermore, if multiple reference pictures are supported, a reference picture index may also be sent that is used to identify which reference picture in the reference picture store (212) the temporal prediction signal came from.

[0098] A mode decision block (214) in the encoder selects the best prediction mode, for example based on a rate-distortion optimization method. This selection may be made after performing spatial and / or temporal prediction. The intra / inter decision may be indicated by, for example, a prediction mode flag. The prediction block is subtracted from the current video block (216) to generate a prediction residual. The prediction residual is decorrelated using a transform (218) and quantization (220). (For some blocks, the encoder may bypass both transform and quantization, in which case the residual may be directly coded without applying a transform or quantization process.) The quantized residual coefficients are inverse quantized (222) and inverse transformed (224) to form a reconstructed residual, which may then be added back to the prediction block (226) to form a reconstructed signal for the CU. Further loop filtering, such as deblocking / SAO (sample adaptive offset) filtering, may be applied (228) to the reconstructed CU to reduce coding artifacts before they are placed in a reference picture store (212) and used to code future video blocks. To form the output video bitstream 230, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all passed to the entropy decoding unit (108) to be further compressed and packed to form a bitstream.

[0099] Figure 2B A block diagram of a block-based video decoder 250 is shown. In the decoder 250, the bitstream is decoded by the decoder elements described below. The video decoder 250 generally performs a decoding pass that is the inverse of the encoding pass, such as Figure 2A Encoder 200 also typically performs video decoding as part of encoding the video data.

[0100] Specifically, the input of the decoder includes a video bitstream 252, which may be generated by the video encoder 200. The video bitstream 252 is first unpacked and entropy decoded at an entropy decoding unit 254 to obtain transform coefficients, motion vectors, and other coding information. The picture partition information indicates how the picture is partitioned. Therefore, the decoder may divide (256) the picture according to the decoded picture partition information. The decoding mode and prediction information are sent to a spatial prediction unit 258 (if intra-frame decoding) or a temporal prediction unit 260 (if inter-frame decoding) to form a prediction block. The residual transform coefficients are transmitted to an inverse quantization unit 262 and an inverse transform unit 264 to reconstruct the residual block. The prediction block and the residual block are then added together at 266 to generate a reconstructed block. The reconstructed block may further undergo loop filtering 268 before being stored in a reference picture store 270 for use in predicting future video blocks.

[0101] The decoded picture 272 may also undergo post-decoding processing (274), such as an inverse color transform (e.g., a transform from YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process (204). The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream. The decoded processed video may be sent to a display device 276. The display device 276 may be a separate device from the decoder 250, or the decoder 250 and the display device 276 may be components of the same device.

[0102] The various methods and other aspects described in this disclosure may be used to modify modules of the video encoder 200 or decoder 250. In addition, the systems and methods disclosed herein are not limited to VVC or HEVC, and may be applied, for example, to other standards and recommendations (whether pre-existing or developed in the future) and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this disclosure may be used alone or in combination.

[0103] Example immersive video decoding

[0104] Some immersive video coding techniques, such as MIV, receive as input a set of views called MVD (standing for Multi-View plus Depth) views. Such content can be computer graphics or natural content. In such content, both texture information and depth information are available for each pixel of each view.

[0105] In some immersive video coding techniques, segmentation information may be provided in the form of an entity map, where each pixel is associated with a given integer value associated to a given object. This allows marking objects of a scene on the encoder side. An example is described in ISO / IEC 23090-12:2021 section "8.3.2.7 Block data unit MIV extended syntax". Such entity map information allows object filtering on the decoder side, for example during the rendering step.

[0106] Figure 5A An example of a texture map for a scene is illustrated. Figure 5B An example of a depth map of a scene is illustrated. Figure 5C An example of an entity graph for a scene is illustrated.

[0107] An example bitstream generated by an immersive content encoder (such as a TMIV encoder) on an input MVD consists of texture and depth tiles packed into atlases, where these tiles have been extracted from the input views. FIG. 6A to FIG. 6B An example of an input MVD is depicted in , and FIG. 7A to FIG. 7BAn example of an input MVD with entities is depicted in . Each tile is part of an input view, and for each pixel of the tile, there is a texture value and a depth value.

[0108] Fig. 6A An example of a texture tile atlas after TMIV encoding of an MVD sequence is illustrated. Figure 6B The corresponding depth block atlas is illustrated.

[0109] Fig. 7A An example of a texture tile atlas after TMIV encoding of selected entities of an MVD sequence is illustrated. Figure 7B The corresponding depth block atlas is illustrated.

[0110] Given the different types and configurations of content capture devices, the camera that captures color and the camera that captures scene geometry may not be located in exactly the same location. For smartphones with a depth sensor or with an RGB plus depth camera ( FIG. 9A to FIG. 9B ), this is the case. Therefore, it is expected that an immersive video decoding system can correctly handle input content with texture information and depth information emitted from different non-co-located cameras.

[0111] Figure 8 The different coordinates generated when the color camera and the depth camera have different positions are schematically illustrated.

[0112] FIG. 9A to FIG. 9B An example of a camera device is illustrated in which the color sensor and the depth sensor have different locations. Fig. 9A A smartphone 902 is shown with a macro-vision RGB camera 904 , a main RGB camera 906 , a depth camera 908 , and a laser autofocus system 910 . Fig. 9B Depth camera system 912 is shown with right imager 914, IR projector 916, left imager 918, and RGB module 920. Example of a depth camera system that may be used in some implementations in the Intel RealSense D435.

[0113] Example embodiments provide an immersive video coding system capable of processing input material that uses non-co-located cameras for texture and depth and also has segmentation information. Example embodiments provide information about which camera (texture or depth) was used to provide the segmentation. This may depend on the image processing that has been used to generate the segmentation, for example, whether contour object detection is performed by depth analysis or by color analysis.

[0114] In the case of non-co-located RGB and depth cameras, there is not necessarily a 1-to-1 bijection between depth patches and texture patches.

[0115] An immersive video encoder such as the TMIV (Test Model for Immersive Video) encoder takes as input one or more MVDs along with segmentation information. The segmentation information may include a list of entities in an entity graph. For example, in an entity graph, all pixels belonging to a given object may be assigned the same integer value, and pixels belonging to different objects may be assigned different integer values. The encoding process generates a tile for each entity to be sent. Since the entities are associated with a depth map or a texture map, the generated tiles are correctly generated for the texture tiles or the depth tiles.

[0116] In the current MIV immersive video specification, the segmented tiles are accompanied by corresponding tile data units (PDUs). Each tile data unit includes a tile data unit entry identifier pdu_entity_id in its PDU. Table 1 shows an example syntax of the PDU, as described in the currently available 23090-12 specification. One problem with this system is that only one tile (depth or texture) is correctly shaped with respect to the segmentation information.

[0117]

[0118] Table 1

[0119] An example of an immersive video rendering process may include some or all of the following stages:

[0120] ●Warping stage: depth de-projection / re-projection from each source view to the target view.

[0121] • Visibility stage: For each pixel of the target view, a depth selection is made from all projected depth values.

[0122] - Filtering stage (optional): Depth map filtering by local pixel neighborhood.

[0123] ●Blending stage: Shading map calculation, blending the warped color values ​​from all source views.

[0124] In the case of rendering only some objects, the incoming data may be filtered based on the number of entities. In the case where there is no 1-to-1 bijection between texture data and depth data, this process should be performed at the appropriate step of rendering with respect to segmentation information. Example embodiments provide encoding and decoding of information indicating which input data is used for segmentation. With respect to filtering, such information may be used as follows.

[0125] - In an example embodiment, if the segmentation information is linked to the depth partition, this information is directly used in the warping stage by only warping the depth pixels belonging to the selected entity.

[0126] - In an example embodiment, if segmentation information is linked to the texture tiles, this information may be used when computing the shading graph (d) to blend only the color values ​​of the distorted color pixels belonging to the selected entity.

[0127] The information about which input data is used for segmentation can be further used in the disambiguation process. As with any rendering, subjective performance is related to the way the contour is rendered. In some embodiments, the information about which map (depth map or texture map) carries the segmentation can be used to disambiguate the contour at the rendering stage.

[0128] ● In case segmentation information is generated using depth data: During the visibility phase, a decision is made about the depth selected for the rendered viewport. For noisy depth maps, ambiguity may arise on some pixels of the contours. In an example embodiment, segmentation information can be used to decide whether a pixel belongs to a foreground object or background, as it adds a second stage selection, such as a majority voting procedure on the entity identifier.

[0129] ● Segmentation information about textures: Given the same ambiguity on some pixels in the visibility phase, in some embodiments, there is no final decision about the visibility map in the visibility phase; instead, a "soft" or temporary decision is made with the potential depth value. Then, during the blending phase, a majority voting procedure based on entity identifiers can be applied to disambiguate and decide on the visibility map.

[0130] In some embodiments, immersive video data may be collected using equipment with two different types of depth sensors. The first sensor is cheap and has a relatively low spatial resolution (a small number of pixels along the width / height) and / or inaccurate depth resolution (on the z-axis). The second sensor may have a finer depth sensor with more pixels and / or higher precision on the z-axis. For example, the equipment can be used to capture a scene with some characters. In some embodiments, segmentation can be performed using only the data from the second type of depth sensor, for example to segment the face of a character, so that only these parts are sent, thereby reducing the overall transmission bit rate. On the reconstruction side, the depth from the lower precision sensor can be used to reconstruct the scene, while the depth from the high precision depth sensor is used to reconstruct the face of the character. For a selected entity, one face can be placed in high resolution, followed by another face, and so on.

[0131] In some embodiments, information indicating whether segmentation information is provided for each camera is provided in the metadata bitstream for that camera. In some embodiments, this information may be carried by a Boolean value, but integer values, floating point values, or other values ​​may alternatively be used.

[0132] Since any tile comes from a view, there is no need to export that information at the tile level.

[0133] Example embodiments may use Fig.10 The camera array layout is schematically depicted in FIG. In this example layout, four depth sensors Vd j ,j∈[1,4] is placed in a Vc with nine color sensors i ,i∈[1,9], is surrounded by a 3×3 color camera arrangement. However, the present invention is not limited to this particular camera arrangement, and is also applicable to other numbers and relative positions of color sensors and depth sensors.

[0134] Fig.11 An example of a complete MIV processing chain from 3D scene acquisition to synthesis of a virtual viewport is depicted in . Spatially unaligned color and depth views are ingested by the MIV encoder as separate input views, each with its own extrinsic and intrinsic camera parameters, and on the decoding side, the renderer is provided with the decoded color and depth views.

[0135] To synthesize a new virtual viewpoint, the renderer backprojects the (u,v) pixels of each decoded view to their (x,y,z) 3D positions and then reprojects their color attributes to the corresponding pixel coordinates (u',v') in the virtual viewport. Fig.12 A schematic diagram of the reprojection process of the source view with a perspective camera model is illustrated in . Since the depth value r(u,v) of the color pixel (u,v) is not directly available, this direct back-projection-reprojection process is not simple in the context where the source depth map is not aligned with the source color map.

[0136] Example Encoded Data

[0137] In an example embodiment, information indicating whether segmentation data is provided for a view is signaled (e.g., in a bitstream) using a flag such as mvp_segmentation_flag[v]. Such a flag may be included, for example, in a MIV view parameter list. The following table illustrates an example syntax of a MIV view parameter list including a flag for each view indicating whether segmentation information is provided for the corresponding view.

[0138]

[0139] Table 2

[0140] Information identifying the original camera (texture or depth) from which the segmentation has been calculated can be used during the rendering phase, for example when making decisions about object outlines.

[0141] Example methods and systems

[0142] like Fig.13 As illustrated, the method performed in some embodiments includes obtaining multiple source views of a scene (1302), for example from one or more color cameras and one or more depth cameras. For at least one of the source views, segmentation information is obtained (1306) that associates each of a plurality of regions of the source view with a respective entity. In this example, the segmentation information for the view is received from the depth camera 1304, although segmentation information for one or more of the other views may be received. The segmentation information may take the form of an entity map, although alternative forms may be used in other embodiments. One or more of a variety of different available techniques may be used to segment the image (1308) into different entities, such as contour object detection using color and / or depth as appropriate.

[0143] In some embodiments, using an enzyme such as E. Depth-based segmentation is performed using techniques such as those described in B. Chupeau, “Depth-Based Segmentation”, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 7, No. 1, February 1997. In some embodiments, segmentation of a scene is based on the color or texture of an image of the scene, such as using the techniques described in J. Malik, S. Belongie, T. Leung, and J. Shi, “Contour and Texture Analysis for Image Segmentation”, International Journal of Computer Vision, Vol. 43, No. 1, pp. 7-27, 2001.

[0144] The multiple source views are encoded into an immersive video including multiple tiles, wherein the tiles are segmented according to the segmentation information. In some embodiments, the encoding includes analyzing multiple views of the scene to remove redundant pixels (1310) that represent portions of the scene viewed by more than one camera of the same type of camera. The remaining pixels are arranged (1312) into tiles in a texture atlas and a depth atlas according to the segmentation information. In some embodiments, each tile includes image information from only one view in the view (other views of the same part of the scene are removed because they are redundant). In order to arrange the tiles according to the segmentation information, the tiles are segmented with the goal that each tile should only include pixels from one of the entities determined according to the segmentation. However, it should be understood that the segmentation will not have perfect accuracy. For tiles from views for which segmentation information is provided, the segmentation may be most accurate, while for tiles from views for which segmentation information is not provided, the segmentation may be less accurate.

[0145] The atlases are encoded (1314) into video frames using a video encoder. Information indicating which source views are associated with the segmentation information used to segment the partition is also encoded (1316). In some embodiments, for each source view, the information indicating which of the source views are associated with the segmentation information used to segment the partition includes a flag indicating whether the input view is associated with the segmentation information used to segment the partition.

[0146] Fig.14 An example implementation of a rendering method is shown in . At 1402, one or more entities of interest may be selected for rendering. This may involve obtaining an identifier for each of the selected entities, such as an identifier pdu_entity_id. In some embodiments, this step is not performed, and all available entities are rendered (provided that they are visible in the scene). At 1404, a depth block and a texture block associated with each of the selected entities of interest are selected from the available blocks in the texture atlas and the depth atlas. At 1406, for each relevant block, information identifying a view that is the source of the block is obtained. This information may be provided in a syntax element such as a block data unit, for example, in a parameter (e.g., pdu_projection_id) in the block data unit syntax structure of Table 3. As shown in Table 3, the block data unit syntax element may be configured to include the pdu_miv_extension of Table 1, which in turn includes the identifier pdu_entity_id, thereby linking each block to the entity identifier of the block and the corresponding view from which the block is obtained.

[0147]

[0148]

[0149] Table 3

[0150] At 1408, information indicating which views are used to provide segmentation information is obtained. Such information may be obtained via a flag associated with each view (such as mvp_segmentation_flag) that indicates whether the associated view is used by the encoder to provide segmentation information. Alternatively, such information may be obtained via other means (e.g., via a list of views used to provide segmentation information). Based on the information associating views with partitions (from 1406) and the information indicating which views are used for segmentation (from 1408), the rendering system determines at 1410 which type of view (texture or depth) is used to provide segmentation information for each partition.

[0151] In some embodiments, where the view providing segmentation information for the tiles is a depth camera view, the rendering process proceeds as follows. At 1412, a depth deprojection / reprojection is performed from each source view to the target view, where the warp is applied only to depth pixels belonging to the selected entity. At 1414, for each pixel of the target view, visibility is determined based on a depth selection from all projected depth values. In some embodiments, optional depth map filtering is performed based on the local neighborhood of each pixel (not shown). At 1416, a shading map calculation is performed, blending the warped color values ​​from all source views.

[0152] In some embodiments, in the case where the view providing segmentation information for the tile is a depth camera view, the rendering process proceeds as follows. At 1418, warping is performed on the depth pixels of the tile. At 1420, at least a temporary visibility determination is made based on the depth value, and the final decision about the visibility map is determined during the subsequent blending stage. As an example, consider N input views, after warping, there are N visibility maps, some of which are completely empty for the current tile being processed. In some embodiments, the temporary visibility determination can retain all of these visibility maps, allowing the final determination of visibility to be made at a later stage. In some embodiments, the rendering system does not retain all N visibility maps, but only keeps track of the minimum depth and maximum depth, which produces two visibility maps instead of N visibility maps. Maintaining more than one visibility map provides flexibility for the following steps of performing the decision. In some embodiments, optional depth map filtering is performed based on the local neighborhood of each pixel (not shown). At 1422, a shading map calculation is performed, blending the warped color values ​​from all source views. At 1424 (which may be concurrent with 1422), a final visibility determination is made. In some implementations, the final determination may be made using a majority voting procedure based on entity identifiers to disambiguate and decide on a visibility map.

[0153] based on Fig.14 During the rendering process, the rendering system may store a scene (e.g., a frame of a video scene) in a frame buffer or otherwise make the scene visible to a user.

[0154] Fig.14 The example of shows a process in which the warping, visibility determination, and blending processes are all different depending on whether a texture view or a depth view is used for segmentation information. However, in some embodiments, one or more of these processes may be the same, regardless of whether texture or depth is used for segmentation.

[0155] Fig.15An example of a data pattern represented in encoded scene data according to some embodiments is illustrated. In this example, the scene includes two entities (entity A and entity B), each of which is identified by an entity identifier (e.g., pdu_entity_id). Four views (texture view 1, texture view 2, depth view 1, depth view 2) are used to capture the entities in this example. Each entity in these views is associated with a flag (e.g., mvp_segmentation_flag) indicating whether the view is used to provide segmentation information. Each tile in the texture atlas 1502 is associated with a texture view in the texture view and is associated with an identifier of the entity represented by the tile. This association can be conveyed via a tile data unit syntax element, as described above. Similarly, each tile in the depth atlas 1504 is associated with a texture view in the texture view and is associated with an identifier of the entity represented by the tile. This information can also be conveyed in a tile data unit syntax element. Although for illustration, Fig.15 The examples of FIG. 4 show a small number of entities, blocks, and views, but it should be understood that some embodiments utilize a greater number of entities, blocks, and views. Fig.15 Using the data mode shown, the rendering system can determine whether segmentation of an entity is performed using a texture view or a depth view, and it can render the entity accordingly.

[0156] In some embodiments, the bitstream is constrained so that the views signaled for partition information are all depth views, or the views signaled for partition information are all texture views. However, in other embodiments, the bitstream is processed to allow handling of the case where some views for partitions are depth views and some views for partitions are texture views. Fig.16 An example method for performing such processing is provided in .

[0157] At 1602, one or more entities of interest may be selected for rendering. This may involve obtaining an identifier for each of the selected entities, such as an identifier pdu_entity_id. In some embodiments, this step is not performed, and all available entities are rendered (provided they are visible in the scene). At 1604, a depth tile and a texture tile associated with each of the selected entities of interest are selected from the available tiles in the texture atlas and the depth atlas. At 1606, for each relevant tile, information identifying a view that is a source for the tile is obtained. This information may be provided in a syntax element such as a tile data unit, for example, in a parameter (e.g., pdu_projection_id) in the tile data unit syntax structure of Table 3 above.

[0158] At 1608, information indicating which views are used to provide segmentation information is obtained. Such information may be obtained via a flag associated with each view (such as mvp_segmentation_flag) that indicates whether the associated view is used by the encoder to provide segmentation information. Alternatively, such information may be obtained by other means (e.g., via a list of views used to provide segmentation information). At 1612, depth deprojection / reprojection is performed from each source view to the target view, where the warp is applied only to depth pixels belonging to the selected entity. At 1614, for each pixel of the target view, an initial visibility determination is made based on a depth selection from the projected depth value according to the block segmented using the depth view. At 1615, the initial visibility determination is refined based on a depth selection from the projected depth value according to the block segmented using the texture view. In some embodiments, optional depth map filtering (not shown) is performed based on the local neighborhood of each pixel. At 1616, initial blending is performed based on the pixels of the texture block segmented using the texture view. At 1617, the initial blend is refined based on pixels of the texture block segmented using the depth view. Fig.16 During the rendering process, the rendering system may store a scene (e.g., a frame of a video scene) in a frame buffer or otherwise make the scene visible to a user. Fig.16 The method is given as an example only. Fig.16 The steps may be performed in various orders in different implementations, and other methods may alternatively be used to perform rendering when both depth views and texture views are used for segmentation.

[0159] Further implementation plans

[0160] The present disclosure describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are specifically described, and at least individual characteristics are shown, usually in a manner that may sound limited. However, this is for clarity in the description and does not limit the disclosure or scope of those aspects. In fact, all different aspects can be combined and interchanged to provide further aspects. In addition, these aspects can also be combined and interchanged with aspects described in previous submissions.

[0161] The aspects described and contemplated in this disclosure can be implemented in many different forms. Although some embodiments are specifically shown, other embodiments are contemplated, and the discussion of specific embodiments does not limit the breadth of specific implementations. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to sending a generated or encoded bitstream. These and other aspects can be implemented as methods, devices, computer-readable storage media having instructions for encoding or decoding video data according to any of the methods, and / or computer-readable storage media having a bitstream generated according to any of the methods stored thereon.

[0162] In this disclosure, the terms "reconstructed" and "decoded" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image", "picture" and "frame" are used interchangeably. Typically, but not necessarily, the term "reconstructed" is used on the encoder side, while "decoded" is used on the decoder side.

[0163] The terms HDR (high dynamic range) and SDR (standard dynamic range) generally convey specific dynamic range values ​​to one of ordinary skill in the art. However, it is also intended that additional implementations be employed in which references to HDR are understood to mean "higher dynamic range" and references to SDR are understood to mean "lower dynamic range." Such additional implementations are not constrained by any specific value of dynamic range that might generally be associated with the terms "high dynamic range" and "standard dynamic range."

[0164] Various methods are described herein, and each method includes one or more steps or actions for implementing the method. Unless the correct operation method requires a specific order of steps or actions, the order and / or purposes of specific steps and / or actions can be modified or combined. In addition, terms such as "first", "second", etc. can be used to modify elements, parts, steps, operations, etc. in various embodiments, such as, for example, "first decoding" and "second decoding". Unless specifically required, the use of such terms does not imply the ordering of the modification operation. Therefore, in this example, the first decoding does not need to be performed before the second decoding, and can, for example, occur before, during, or in overlapping time periods of the second decoding.

[0165] For example, various numerical values ​​may be used in the present disclosure. Specific values ​​are for illustrative purposes, and the aspects are not limited to these specific values.

[0166] The embodiments described herein may be performed by a processor or other hardware or computer software implemented by a combination of hardware and software. As a non-limiting example, these embodiments may be implemented by one or more integrated circuits. As a non-limiting example, the processor may be any type suitable for the technical environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0167] Various implementations involve decoding. As used in the present disclosure, "decoding" may encompass, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such processes include one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also include or alternatively include processes performed by decoders of various implementations described in the present disclosure, such as extracting a picture from a tiled (packed) picture, determining an upsampling filter to be used, then upsampling the picture, and flipping the picture back to its intended orientation.

[0168] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or generally to a broader decoding process will be discerned based on the context of the specific description.

[0169] Various implementations involve encoding. In a manner similar to the discussion above about "decoding", "encoding" as used in the present disclosure may encompass, for example, all or part of a process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential encoding, transforms, quantization, and entropy encoding. In various embodiments, such processes also include or alternatively include processes performed by encoders of various implementations described in the present disclosure.

[0170] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in yet another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or generally to a broader encoding process will be discernible based on the context of the specific description.

[0171] When the figures are presented as flow charts, it should be understood that they also provide block diagrams of corresponding devices. Similarly, when the figures are presented as block diagrams, it should be understood that they also provide flow charts of corresponding methods / processes.

[0172] Each embodiment refers to rate-distortion optimization. Specifically, during the encoding process, the balance or trade-off between rate and distortion is usually considered, which often takes into account the constraints of computational complexity. Rate-distortion optimization is usually expressed as minimizing the rate-distortion function, which is the weighted sum of rate and distortion. There are different methods to solve the rate-distortion optimization problem. For example, these methods can be based on extensive testing of all coding options (including all considered modes or decoding parameter values), and fully evaluate their decoding costs and the related distortion of the reconstructed signal after decoding and decoding. Faster methods can also be used to reduce coding complexity, especially for the calculation of approximate distortion based on prediction or prediction residual signal rather than reconstructed residual signal. A mixture of these two methods can also be used, such as by using approximate distortion only for some possible coding options, and using complete distortion for other coding options. Other methods only evaluate a subset of possible coding options. More generally, many methods use any of various techniques to perform optimization, but optimization is not necessarily a complete evaluation of both decoding cost and related distortion.

[0173] The specific implementations and aspects described herein may be implemented in, for example, methods or processes, devices, software programs, data streams, or signals. Even if discussed only in the context of a single form of specific implementation (e.g., discussed only as a method), the specific implementation of the features discussed may also be implemented in other forms (e.g., devices or programs). The device may be implemented in, for example, appropriate hardware, software, and firmware. The method may be implemented in, for example, a processor generally referring to a processing device, which includes, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, such as, for example, a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate the communication of information between end users.

[0174] Reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearance of the phrase "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations thereof in various places throughout this disclosure are not necessarily all referring to the same embodiment.

[0175] Additionally, the present disclosure may refer to “determining” various pieces of information. Determining information may include, for example, one or more of estimating the information, calculating the information, predicting the information, or retrieving the information from a memory.

[0176] Additionally, the present disclosure may refer to "accessing" various pieces of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0177] Furthermore, the present disclosure may refer to "receiving" various pieces of information. Like "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" generally involves, in one way or another, during operations such as, for example, storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0178] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such phrases are intended to encompass selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first listed option and the second listed option (A and B), or only the first listed option and the third listed option (A and C), or only the second listed option and the third listed option (B and C), or all three options (A and B and C). This can be extended to as many items as listed.

[0179] Moreover, as used herein, the term "signaling" refers to (among other things) indicating something to a corresponding decoder. For example, in certain embodiments, the encoder signals a region-based filter parameter selection for a specific parameter among a plurality of parameters for de-artifact filtering. Thus, in one embodiment, the same parameters are used on both the encoder side and the decoder side. Therefore, for example, the encoder can send (explicit signaling) a specific parameter to the decoder so that the decoder can use the same specific parameter. On the contrary, if the decoder already has specific parameters and other parameters, signaling can be used without sending (implicit signaling) to simply allow the decoder to know and select specific parameters. By avoiding sending any actual function, bit saving is achieved in various embodiments. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more grammatical elements, flags, etc. are used to signal information to the corresponding decoder. Although the verb form "signaling" of the term is previously related to, the term can also be used as a noun "signal" herein.

[0180] Specific implementations may generate various signals, formatting these signals to carry information that can be, for example, stored or transmitted. The information may include, for example, instructions for executing the method or data generated by one of the described specific implementations. For example, the signal may be formatted to carry a bit stream of the described embodiment. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, the signal may be sent via a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

[0181] A number of embodiments are described. Features of these embodiments may be provided individually or in any combination in various claim categories and types. Further, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination in various claim categories and types:

[0182] - A bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0183] - A bitstream or signal comprising syntax conveying information generated according to any of the described embodiments.

[0184] ● Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0185] ●Creation and / or transmission and / or reception and / or decoding according to any one of the embodiments described.

[0186] ●A method, process, device, medium storing instructions, medium storing data or signal according to any one of the embodiments described in the embodiments.

[0187] ●A television, set-top box, mobile phone, tablet computer, head-mounted display, or other electronic device that receives a signal including an encoded image (e.g., over the air or via a wired connection) and performs adaptation of filter parameters according to any of the described embodiments.

[0188] It should be noted that the various hardware elements of one or more of the described embodiments are referred to as "modules" that perform (i.e., execute, implement, etc.) the various functions described herein in conjunction with the corresponding modules. As used herein, a module includes hardware that is considered suitable for a given specific implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices). Each of the modules described may also include executable instructions for performing one or more functions described as being performed by the corresponding module, and it should be noted that these instructions may take the form of or include the following instructions: hardware (i.e., hard-wired) instructions, firmware instructions, software instructions, etc., and may be stored in any suitable one or more non-transitory computer-readable media (such as commonly referred to as RAM, ROM, etc.).

[0189] Although features and elements are described above in particular combinations, each feature or element may be used alone or in any combination with other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media (such as built-in hard disks and removable disks), magneto-optical media, and optical media (such as CD-ROM disks and digital versatile disks (DVDs)). A processor associated with software may be used to implement a radio frequency transceiver for a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method, comprising: Get multiple source views of the scene; For at least one of the source views, obtaining segmentation information associating each of a plurality of regions of the source view with a corresponding entity; encoding the plurality of source views into an immersive video comprising a plurality of partitions, the partitions being partitioned according to the partition information; as well as Information indicating which of the source views are associated with the partitioning information used to partition the partition is encoded.

2. An apparatus comprising one or more processors, wherein the one or more processors are configured to perform the following operations: Get multiple source views of the scene; For at least one of the source views, obtaining segmentation information associating each of a plurality of regions of the source view with a corresponding entity; encoding the plurality of source views into an immersive video comprising a plurality of partitions, the partitions being partitioned according to the partition information; as well as Information indicating which of the source views are associated with the partitioning information used to partition the partition is encoded.

3. The method according to claim 1 or the device according to claim 2, wherein, for each source view, the information indicating which of the source views are associated with the segmentation information for segmenting the block includes a flag indicating whether the input view is associated with the segmentation information for segmenting the block. 4 . The method of claim 1 or the apparatus of claim 2 , wherein the segmentation information associated with a source view comprises an entity graph associated with the source view.

5. The method according to claim 1 or the device according to claim 2, wherein the information indicating which of the source views are associated with the segmentation information used to segment the block includes information indicating whether the segmentation information is based on a depth image or a texture image.

6. A method comprising: Obtaining an encoded immersive video, the encoded immersive video comprising a plurality of segments, the video representing a plurality of input views of a scene; obtaining information indicating which of the input views are associated with segmentation information used to segment the partition; as well as The immersive video is rendered according to the information indicating which of the input views are associated with the segmentation information used to segment the tiles.

7. An apparatus comprising one or more processors, wherein the one or more processors are configured to perform the following operations: Obtaining an encoded immersive video, the encoded immersive video comprising a plurality of segments, the video representing a plurality of input views of a scene; obtaining information indicating which of the input views are associated with segmentation information used to segment the partition; as well as The immersive video is rendered according to the information indicating which of the input views are associated with the segmentation information used to segment the tiles.

8. The method according to claim 6 or the apparatus according to claim 7, wherein, for each source view, the information indicating which of the source views are associated with the segmentation information for segmenting the block comprises a flag indicating whether the input view is associated with the segmentation information for segmenting the block.

9. The method according to claim 6 or the device according to claim 7, wherein the information indicating which of the source views are associated with the segmentation information used to segment the block includes information indicating whether the segmentation information is based on a depth image or a texture image.

10. The method of claim 6 or the apparatus of claim 7, wherein rendering the immersive video according to the information indicating which of the input views are associated with the segmentation information for segmenting the tiles comprises: identifying at least one selected entity to render; as well as In response to determining that the segmentation information is based on a depth image, warping of depth pixels of the immersive video is performed only for depth tiles associated with the at least one selected entity.

11. The method of claim 6 or the apparatus of claim 7, wherein rendering the immersive video according to the information indicating which of the input views are associated with the segmentation information for segmenting the tiles comprises: identifying at least one selected entity to render; as well as In response to determining that the segmentation information is based on the texture image: performing warping of depth pixels of the immersive video for depth partitions including at least a depth partition associated with the at least one selected entity; as well as The blending of color values ​​is performed based only on colored pixels associated with the at least one selected entity.

12. The method of claim 6 or the apparatus of claim 7, wherein rendering the immersive video according to the information indicating which of the input views are associated with the segmentation information for segmenting the tiles comprises: In response to determining that the segmentation information is based on a depth image, a visibility determination is made based at least in part on the segmentation information.

13. The method of claim 6 or the apparatus of claim 7, wherein rendering the immersive video according to the information indicating which of the input views are associated with the segmentation information for segmenting the tiles comprises: In response to determining that the segmentation information is based on a texture image, a visibility determination is made during a blending stage based at least in part on the segmentation information.

14. A signal, comprising: an encoded immersive video, the encoded immersive video comprising a plurality of tiles, the video representing a plurality of input views of a scene; and Information indicating which of the input views are associated with segmentation information used to segment the partition.

15. The signal of claim 14, wherein, for each source view, the information indicating which of the source views are associated with the partitioning information for partitioning the partition comprises a flag indicating whether the input view is associated with the partitioning information for partitioning the partition.

16. A computer readable medium storing the signal according to claim 14.

17. A computer readable medium comprising instructions for causing one or more processors to perform the method of any one of claims 1, 3 to 6, or 8 to 13.

18. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1, 3 to 6 or 8 to 13.

Images