Signaling Volumetric Visual Video-Based Coding Content in Immersive Scene Descriptions

Abstract

The present systems and methods are directed to volumetric video representations in scene descriptions. The described systems and methods present a generic design for supporting V3C / V-PCC content. The systems and methods include semantics and a binary representation for decoded V3C / V-PCC media data in a buffer format that can be used by a presentation engine for reconstruction and rendering.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 63 / 299,874, filed January 14, 2022, U.S. Provisional Patent Application No. 63 / 326,628, filed April 1, 2022, U.S. Provisional Patent Application No. 63 / 355,936, filed June 27, 2022, U.S. Provisional Patent Application No. 63 / 417,225, filed October 18, 2022, and U.S. Provisional Patent Application No. 63 / 417,967, filed October 20, 2022, the contents of which are incorporated herein by reference. [Background technology]

[0002] A rendering / presentation engine may be used to compose and render a scene. The information used to compose and render a scene may be provided by a scene description. The Moving Picture Experts Group (MPEG) has incorporated glTF as a scene graph format and is extending glTF to support dynamic (timed) content such as 2D and 360-degree video, dynamic volumetric visual media, and audio. Additional support for volumetric visual media in glTF via extensions may be desired.

[0003] An Exploration Experiment (EE) has been initiated to support MPEG immersive codecs in MPEG scene descriptions. The EE is targeted at devising design principles and workflows for immersive media coded content using MPEG technologies such as video-based point cloud compression (V-PCC), geometry-based point cloud compression (G-PCC), and MPEG immersive media (MIV). Efforts are being made to specify how to play volumetric visual media that is part of a glTF scene. Two different approaches are being investigated as possible technologies to support volumetric visual video-based coding (V3C) content. However, these approaches are limited and do not accommodate different types of V3C coded content. Summary of the Invention

[0004] The present systems and methods are directed to volumetric video representation in scene descriptions. The described systems and methods present a generic design for supporting V3C / V-PCC content. The systems and methods include semantics and a binary representation for decoded V3C / V-PCC media data in a buffer format that can be used by a presentation engine for scene composition / reconstruction and rendering.

[0005] A glTF extension that supports and enables the use of the YCbCr texture format in MPEG-I scene descriptions. The extension adds to the properties of YCbCr textures. A process is described that allows appropriate shader implementations or native GPU implementations to use the information in the extension. The extension adapts the YCbCr texture descriptions used in modern graphics APIs such as Vulkan.

[0006] A glTF extension to support timed 3D video in MPEG-I scene descriptions coded using V3C (ISO / IEC 23090-5) techniques. Such timed 3D video can be V-PCC (as enumerated in ISO / IEC 23090-5) and MIV (ISO / IEC 23090-12).

[0007] A semantic mechanism is introduced, by which the decoded V3C components are represented in the MPEG-I Scene Description. Each V3C component is represented as a JSON literal under the glTF extensions. Some of the components have specific properties that are further described in the respective components, namely V3C Attributes Component and V3C Atlas Component. Furthermore, a processing method is described for the presentation engine that utilizes the information stored under the glTF extensions.

[0008] A buffer format for the decoded atlas data provides the relevant information to reconstruct a representation of 3D video from the decoded 2D video stream. The buffer formats can be of different types, each type providing relevant application-specific information. In particular for MIV, the use of common atlas data to include view parameters is also expressed as a property in the glTF extension. Buffer formats that store the view parameters are exchanged with the presentation engine, thereby allowing different views to be reconstructed. [Brief description of the drawings]

[0009] A more detailed understanding may be had from the following description, given by way of example in conjunction with the accompanying drawings, in which like reference numbers indicate similar elements and in which: [Figure 1A] FIG. 1A is a system diagram illustrating an example communication system in which one or more disclosed embodiments may be implemented. [Figure 1B] FIG. 1B is a system diagram illustrating an example wireless transmit / receive unit (WTRU) that may be used within the communications system illustrated in FIG. 1A, according to one embodiment. [Figure 1C] FIG. 1C is a system diagram illustrating an example radio access network (RAN) and an example core network (CN) that may be used within the communication system illustrated in FIG. 1A, according to one embodiment. [Figure 1D] FIG. 1D is a system diagram illustrating a further exemplary RAN and a further exemplary CN that may be used within the communications system illustrated in FIG. 1A, according to one embodiment. [Diagram 2] FIG. 2 illustrates an exemplary MPEG scene description architecture. [Diagram 3] FIG. 3 illustrates an example pipeline design for processing V-PCC content. [Figure 4] FIG. 4 illustrates an exemplary V3C profile. [Diagram 5] FIG. 5 illustrates an example atlas data representation for patch types PROJECTED and PLR that are enabled at the patch level. [Figure 6-1] FIG. 6-1 illustrates multiple accessors accessing different sub-blocks within the atlas binary data. [Figure 6-2] FIG. 6-2 illustrates multiple accessors accessing different sub-blocks within the atlas binary data. [Figure 7-1] FIG. 7-1 illustrates an atlas data representation that includes data for patch type PROJECTED and PLR information available at the block level. [Figure 7-2] FIG. 7-2 illustrates an atlas data representation that includes data for patch type PROJECTED and PLR information available at the block level. [Figure 8-1] FIG. 8-1 illustrates the representation of sub-blocks and how the atlas data is read. [Figure 8-2] FIG. 8-2 illustrates the representation of sub-blocks and how the atlas data is read. [Figure 9-1] Figure 9-1 illustrates a method for media workflow of V3C content. [Figure 9-2] Figure 9-2 illustrates a method for media workflow of V3C content. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] 1A is a diagram illustrating an example communications system 100 in which one or more disclosed embodiments may be implemented. Communications system 100 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcasts, etc., to multiple wireless users. Communications system 100 may enable multiple wireless users to access such content through sharing of system resources, including wireless bandwidth. For example, the communications system 100 may use one or more channel access methods such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word discrete Fourier transform Spread OFDM (ZT-UW-DFT-S-OFDM), unique word OFDM (UW-OFDM), resource block filtered OFDM, filter bank multicarrier (FBMC), and the like.

[0011] 1A, the communications system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, a radio access network (RAN) 104, a core network (CN) 106, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, although it may be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d, any of which may be referred to as a station (STA), may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, landline or mobile subscriber units, subscription-based units, pagers, mobile phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearables, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain contexts), home electronic devices, devices operating in commercial and / or industrial wireless networks, etc. Any of the WTRUs 102a, 102b, 102c, and 102d may be interchangeably referred to as a UE.

[0012] The communications system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communications networks, such as the CN 106, the Internet 110, and / or other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node B, an eNodeB (eNB), a Home Node B, a home eNodeB, a next generation Node B (e.g., gNodeB (gNB)), a new radio (NR) Node B, a site controller, an access point (AP), a wireless router, etc. Although the base stations 114a, 114b are each depicted as a single element, it can be understood that the base stations 114a, 114b can include any number of interconnected base stations and / or network elements.

[0013] The base station 114a may be part of the RAN 104, which may also include other base stations, such as a base station controller (BSC), a radio network controller (RNC), relay nodes, and / or network elements (not shown). The base station 114a and / or the base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide wireless service coverage for a particular geographic area, which may be relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e., one transceiver for each sector of the cell. In one embodiment, the base station 114a may employ multiple-input multiple output (MIMO) technology and may utilize multiple transceivers for each sector of the cell, for example, using beamforming to transmit and / or receive signals in a desired spatial direction.

[0014] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d via an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0015] More specifically, as noted above, the communications system 100 may be a multiple access system, but may use one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, the base stations 114a and the WTRUs 102a, 102b, 102c of the RAN 104 may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 116 using wideband CDMA (WCDMA). WCDMA may include communications protocols such as High-Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High-Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High-Speed ​​Uplink (UL) Packet Access (HSUPA).

[0016] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro).

[0017] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as NR radio access, which may establish the air interface 116 using NR.

[0018] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may implement LTE radio access and NR radio access together, e.g., using dual connectivity (DC) principles. Thus, the air interface utilized by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to and from multiple types of base stations (e.g., eNBs and gNBs).

[0019] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement a wireless technology such as IEEE 802.11 (i.e., Wireless Fidelity, WiFi), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access, WiMAX), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), or the like.

[0020] 1A may be, for example, a wireless router, a Home NodeB, a Home eNodeB, or an access point, and may utilize any suitable RAT to facilitate wireless connectivity in a localized area, such as an office, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a road, etc. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology, such as IEEE 802.11, to establish a wireless local area network (WLAN). In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology, such as IEEE 802.15, to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may establish a picocell or a femtocell using a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.). As shown in FIG. 1A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not need to access the Internet 110 through the CN 106.

[0021] The RAN 104 may communicate with the CN 106, which may be any type of network configured to provide voice, data, application, and / or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have various quality of service (QoS) requirements, such as, for example, different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. The CN 106 may provide call control, billing services, mobile location-based services, prepaid calling, Internet connectivity, video distribution, etc., and / or perform high-level security functions such as user authentication. Although not shown in FIG. 1A, it may be understood that the RAN 104 and / or the CN 106 may communicate directly or indirectly with other RANs that use the same RAT as the RAN 104 or a different RAT. For example, in addition to being connected to the RAN 104, which may utilize NR radio technology, the CN 106 may also communicate with another RAN (not shown) employing GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0022] The CN 106 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network providing plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the transmission control protocol (TCP), the user datagram protocol (UDP), and / or the internet protocol (IP) of the TCP / IP Internet protocol suite. The networks 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs that may use the same RAT as the RAN 104 or a different RAT.

[0023] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links). For example, the WTRU 102c shown in FIG. 1A may be configured to communicate with a base station 114a, which may employ a cellular-based wireless technology, and a base station 114b, which may employ an IEEE 802 wireless technology.

[0024] Figure 1B is a system diagram illustrating an example WTRU 102. As shown in Figure 1B, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138. It may be understood that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0025] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. Although FIG. 1B depicts the processor 118 and the transceiver 120 as separate components, it may be understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0026] The transmit / receive element 122 may be configured to transmit or receive signals to or from a base station (e.g., base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In one embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF and light signals. It may be understood that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0027] 1B as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0028] The transceiver 120 may be configured to modulate signals transmitted by the transmit / receive element 122 and demodulate signals received by the transmit / receive element 122. As mentioned above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as, for example, NR and IEEE 802.11.

[0029] The processor 118 of the WTRU 102 may be coupled to and may receive user-entered data from a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light-emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from and store data in any type of suitable memory, such as non-removable memory 130 and / or removable memory 132. The non-removable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, etc. In other embodiments, the processor 118 may access information from and store data in memory that is not physically located on the WTRU 102, such as on a server or home computer (not shown).

[0030] The processor 118 may receive power from the power source 134 and may be configured to distribute and / or control the power to other components in the WTRU 102. The power source 134 may be any suitable device for providing power to the WTRU 102. For example, the power source 134 may include one or more dry batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0031] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or in lieu of, information from the GPS chipset 136, the WTRU 102 may receive location information from a base station (e.g., base stations 114a, 114b) over the air interface 116 and / or determine its location based on the timing of signals being received from two or more nearby base stations. It may be understood that the WTRU 102 may obtain location information by way of any suitable location determination method while remaining consistent with an embodiment.

[0032] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or videos), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, and the like. The peripherals 138 may include one or more sensors. The sensor may be one or more of a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor, a geolocation sensor, an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, a humidity sensor, and the like.

[0033] The WTRU 102 may include a full-duplex radio where the transmission and reception of some or all of the signals (e.g., associated with a particular subframe of both the UL (e.g., for transmission) and DL (e.g., for reception)) may be simultaneous and / or together. The full-duplex radio may include an interference management unit to reduce and or substantially eliminate self-interference via either hardware (e.g., a choke) or signal processing via a processor (e.g., via a separate processor (not shown) or via processor 118). In one embodiment, the WTRU 102 may include a half-duplex radio where the transmission and reception of some or all of the signals (e.g., associated with a particular subframe of either the UL (e.g., for transmission) or DL ​​(e.g., for reception)) may be simultaneous and / or together.

[0034] 1C is a system diagram illustrating the RAN 104 and the CN 106, according to one embodiment. As mentioned above, the RAN 104 may employ E-UTRA radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 104 may also communicate with the CN 106.

[0035] The RAN 104 may include eNodeBs 160a, 160b, 160c, although it may be understood that the RAN 104 may include any number of eNodeBs while remaining consistent with an embodiment. The eNodeBs 160a, 160b, 160c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In an embodiment, the eNodeBs 160a, 160b, 160c may implement MIMO technology. Thus, the eNodeB 160a may, for example, use multiple antennas to transmit wireless signals to and / or receive wireless signals from the WTRU 102a.

[0036] Each of the eNodeBs 160a, 160b, 160c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the UL and / or DL, etc. As shown in FIG 1C, the eNodeBs 160a, 160b, 160c may communicate with one another via an X2 interface.

[0037] 1C may include a mobility management entity (MME) 162, a serving gateway (SGW) 164, and a packet data network (PDN) gateway (PGW) 166. Although the foregoing elements are depicted as part of the CN 106, it may be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0038] The MME 162 may be connected to each of the eNodeBs 162a, 162b, 162c in the RAN 104 via an S1 interface and may function as a control node. For example, the MME 162 may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, activating / deactivating bearers, selecting a particular serving gateway during initial attachment of the WTRUs 102a, 102b, 102c, etc. The MME 162 may provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as GSM and / or WCDMA.

[0039] The SGW 164 may be connected to each of the eNodeBs 160a, 160b, 160c in the RAN 104 via an S1 interface. The SGW 164 may generally route and forward user data packets to and from the WTRUs 102a, 102b, 102c. The SGW 164 may perform other functions, such as anchoring the user plane during inter-eNodeB handover, triggering paging when DL data is available to the WTRUs 102a, 102b, 102c, and managing and storing the context of the WTRUs 102a, 102b, 102c.

[0040] The SGW 164 may be connected to a PGW 166, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices.

[0041] The CN 106 may facilitate communications with other networks. For example, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices. For example, the CN 106 may include or communicate with an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that serves as an interface between the CN 106 and the PSTN 108. In addition, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers.

[0042] Although the WTRU is depicted in FIGS. 1A-1D as a wireless terminal, in certain representative embodiments, it is contemplated that such a terminal may use a wired communications interface (e.g., temporarily or permanently) with the communications network.

[0043] In an exemplary embodiment, the other network 112 may be a WLAN.

[0044] A WLAN in infrastructure Basic Service Set (BSS) mode may have an Access Point (AP) of the BSS and one or more stations (STAs) associated with the AP. The AP may have access or interface to a Distribution System (DS) or another type of wired / wireless network that carries traffic within the BSS and / or outside the BSS. Traffic originating from outside the BSS to the STA may arrive through the AP and be delivered to the STA. Traffic originating from the STA to a destination outside the BSS may be sent to the AP to be delivered to the respective destination. Traffic between STAs within the BSS may be sent, for example, through the AP, where the source STA may send traffic to the AP, which may deliver the traffic to the destination STA. Traffic between STAs within the BSS may be considered and / or referred to as peer-to-peer traffic. Peer-to-peer traffic may be sent between (e.g., directly between) the source STA and the destination STA using a direct link setup (DLS). In certain representative embodiments, the DLS may use 802.11e DLS or 802.11z tunneled DLS (TDLS). A WLAN using an Independent BSS (IBSS) mode may not have an AP, and STAs (e.g., all of the STAs) within or using the IBSS may communicate directly with each other. The IBSS mode of communication may be referred to herein as an "ad-hoc" communication mode.

[0045] When using an 802.11ac infrastructure mode of operation or a similar mode of operation, an AP may transmit a beacon on a fixed channel, such as a primary channel. The primary channel may be a fixed width (e.g., a 20 MHz wide bandwidth) or a dynamically configured width. The primary channel may be the operating channel of the BSS, but may be used by STAs to establish a connection with the AP. In certain representative embodiments, Carrier Sense Multiple Access with Collision Avoidance (CSMA / CA) may be implemented, for example, in an 802.11 system. With CSMA / CA, STAs (e.g., all STAs), including the AP, may sense the primary channel. If the primary channel is sensed / detected and / or determined to be busy by a particular STA, the particular STA may back off. One STA (e.g., only one station) may transmit at any given time in a given BSS.

[0046] A High Throughput (HT) STA may use a 40 MHz wide channel for communication, which may be formed, for example, through a combination of a primary 20 MHz channel and adjacent or non-adjacent 20 MHz channels.

[0047] A Very High Throughput (VHT) STA may support 20 MHz, 40 MHz, 80 MHz, and / or 160 MHz wide channels. A 40 MHz and / or 80 MHz channel may be formed by combining multiple contiguous 20 MHz channels. A 160 MHz channel may be formed by combining eight contiguous 20 MHz channels or by combining two non-contiguous 80 MHz channels, which may be referred to as an 80+80 configuration. For the 80+80 configuration, after channel encoding, the data may pass through a segment parser that may split the data into two streams. Inverse Fast Fourier Transform (IFFT) processing and time domain processing may be performed separately on each stream. The streams may be mapped to two 80 MHz channels and the data may be transmitted by the transmitting STA. At the receiver of the receiving STA, the operations described above for the 80+80 configuration may be reversed and the combined data may be sent to the Medium Access Control (MAC).

[0048] Sub-1 GHz operation modes are supported by 802.11af and 802.11ah. The channel operating bandwidths and carriers are reduced in 802.11af and 802.11ah compared to those used in 802.11n and 802.11ac. 802.11af supports 5 MHz, 10 MHz, and 20 MHz bandwidths in the TV White Space (TVWS) spectrum, while 802.11ah supports 1 MHz, 2 MHz, 4 MHz, 8 MHz, and 16 MHz bandwidths using non-TVWS spectrum. According to a representative embodiment, 802.11ah may support meter-type control / machine-type communications (MTC), such as MTC devices in macro coverage areas. MTC devices may have limited capabilities, including certain capabilities, for example, support for (e.g., support only for) certain and / or limited bandwidths. The MTC device may include a battery that has a battery life above a threshold (eg, to maintain a very long battery life).

[0049] WLAN systems that may support multiple channels and channel bandwidths, such as 802.11n, 802.11ac, 802.11af, and 802.11ah, include a channel that may be designated as a primary channel. The primary channel may have a bandwidth equal to the maximum common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel may be configured and / or limited by the STAs among all STAs operating in the BSS that support the smallest bandwidth operating mode. In an 802.11ah embodiment, the primary channel may be 1 MHz wide for STAs (e.g., MTC-type devices) that support (e.g., only) the 1 MHz mode, even if the AP and other STAs in the BSS support 2 MHz, 4 MHz, 8 MHz, 16 MHz, and / or other channel bandwidth operating modes. Carrier sensing and / or Network Allocation Vector (NAV) configuration may depend on the status of the primary channel. For example, if a STA (that only supports 1 MHz mode of operation) transmitting to an AP has a busy primary channel, all of the available frequency bands may be considered busy even if most of the available frequency bands are idle.

[0050] In the United States, the available frequency bands that can be used by 802.11ah are 902MHz to 928MHz. In South Korea, the available frequency bands are 917.5MHz to 923.5MHz. In Japan, the available frequency bands are 916.5MHz to 927.5MHz. The total bandwidth available for 802.11ah is 6MHz to 26MHz depending on the country code.

[0051] 1D is a system diagram illustrating the RAN 104 and the CN 106, according to one embodiment. As mentioned above, the RAN 104 may use NR radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 104 may also communicate with the CN 106.

[0052] The RAN 104 may include gNBs 180a, 180b, 180c, although it may be understood that the RAN 104 may include any number of gNBs while remaining consistent with an embodiment. The gNBs 180a, 180b, 180c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In an embodiment, the gNBs 180a, 180b, 180c may implement MIMO technology. For example, the gNBs 180a, 108b may utilize beamforming to transmit signals to and / or receive signals from the gNBs 180a, 180b, 180c. Thus, the gNB 180a may transmit and / or receive wireless signals to and from the WTRU 102a, for example, using multiple antennas. In one embodiment, the gNBs 180a, 180b, 180c may implement carrier aggregation technology. For example, the gNB 180a may transmit multiple component carriers to the WTRU 102a (not shown). A subset of these component carriers may be on an unlicensed spectrum, while the remaining component carriers may be on a licensed spectrum. In one embodiment, the gNBs 180a, 180b, 180c may implement Coordinated Multi-Point (CoMP) technology. For example, the WTRU 102a may receive coordinated transmissions from the gNBs 180a and 180b (and / or gNB 180c).

[0053] The WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using transmissions associated with scalable numerology. For example, the OFDM symbol spacing and / or OFDM subcarrier spacing may vary for different transmissions, different cells, and / or different portions of the wireless transmission spectrum. The WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using subframes or transmission time intervals (TTIs) of varying or scalable lengths (e.g., including varying numbers of OFDM symbols and / or varying lengths of absolute time duration and varying times).

[0054] The gNBs 180a, 180b, 180c may be configured to communicate with the WTRUs 102a, 102b, 102c in a standalone configuration and / or a non-standalone configuration. In a standalone configuration, the WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c without accessing another RAN (e.g., eNodeBs 160a, 160b, 160c, etc.). In a standalone configuration, the WTRUs 102a, 102b, 102c may utilize one or more of the gNBs 180a, 180b, 180c as mobility anchor points. In a standalone configuration, the WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using signals in unlicensed bands. In a non-standalone configuration, the WTRUs 102a, 102b, 102c may communicate with and connect to a gNB 180a, 180b, 180c while also communicating with and connecting to another RAN, such as an eNodeB 160a, 160b, 160c. For example, the WTRUs 102a, 102b, 102c may implement DC principles to communicate with one or more gNBs 180a, 180b, 180c and one or more eNodeBs 160a, 160b, 160c substantially simultaneously. In a non-standalone configuration, the eNodeBs 160a, 160b, 160c may act as mobility anchors for the WTRUs 102a, 102b, 102c, and the gNBs 180a, 180b, 180c may provide additional coverage and / or throughput for serving the WTRUs 102a, 102b, 102c.

[0055] Each of the gNBs 180a, 180b, 180c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in UL and / or DL, support for network slicing, DC, interworking between NR and E-UTRA, routing of user plane data to User Plane Functions (UPFs) 184a, 184b, routing of control plane information to Access and Mobility Management Functions (AMFs) 182a, 182b, etc. As shown in FIG. 1D, the gNBs 180a, 180b, 180c may communicate with each other via an Xn interface.

[0056] 1D may include at least one AMF 182a, 182b, at least one UPF 184a, 184b, at least one Session Management Function (SMF) 183a, 183b, and possibly a Data Network (DN) 185a, 185b. Although the foregoing elements are depicted as part of the CN 106, it may be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0057] The AMF 182a, 182b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 104 via an N2 interface and may function as a control node. For example, the AMF 182a, 182b may be responsible for user authentication of the WTRUs 102a, 102b, 102c, support for network slicing (e.g., handling different protocol data unit (PDU) sessions having different requirements), selection of a particular SMF 183a, 183b, management of registration areas, termination of non-access stratum (NAS) signaling, mobility management, etc. The network slicing may be used by the AMF 182a, 182b to customize the CN support of the WTRUs 102a, 102b, 102c based on the type of service utilizing the WTRUs 102a, 102b, 102c. For example, different network slices may be established for different use cases, such as services relying on ultra-reliable low latency (URLLC) access, services relying on enhanced massive mobile broadband (eMBB) access, services for MTC access, etc. The AMF 182a, 182b may provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as LTE, LTE-A, LTE-A Pro, and / or non-3GPP access technologies, such as WiFi.

[0058] The SMFs 183a, 183b may be connected to the AMFs 182a, 182b in the CN 106 via an N11 interface. The SMFs 183a, 183b may also be connected to the UPFs 184a, 184b in the CN 106 via an N4 interface. The SMFs 183a, 183b may select and control the UPFs 184a, 184b and configure the routing of traffic through the UPFs 184a, 184b. The SMFs 183a, 183b may perform other functions, such as managing and assigning UE IP addresses, managing PDU sessions, controlling policy enforcement and QoS, providing DL data notification, etc. The PDU session type may be IP-based, non-IP-based, Ethernet-based, etc.

[0059] The UPFs 184a, 184b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 104 via an N3 interface, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices. The UPFs 184, 184b may perform other functions, such as routing and forwarding packets, enforcing user plane policy, supporting multi-homed PDU sessions, handling user plane QoS, buffering DL packets, providing mobility anchoring, etc.

[0060] The CN 106 may facilitate communication with other networks. For example, the CN 106 may include or communicate with an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) that serves as an interface between the CN 106 and the PSTN 108. In addition, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers. In one embodiment, the WTRUs 102a, 102b, 102c may be connected to the local DNs 185a, 185b through the UPFs 184a, 184b via an N3 interface to the UPFs 184a, 184b and an N6 interface between the UPFs 184a, 184b and the DNs 185a, 185b.

[0061] 1A-1D and the corresponding description thereof, one or more or all of the functions described herein with respect to one or more of the WTRUs 102a-102d, base stations 114a-114b, eNodeBs 160a-160c, MME 162, SGW 164, PGW 166, gNBs 180a-180c, AMFs 182a-182b, UPFs 184a-184b, SMFs 183a-183b, DNs 185a-185b, and / or any other devices described herein may be performed by one or more emulation devices (not shown). The emulation devices may be one or more devices configured to emulate one or more or all of the functions described herein. For example, the emulation devices may be used to test other devices and / or to simulate network and / or WTRU functions.

[0062] The emulation device may be designed to implement one or more tests of other devices in a lab environment and / or an operator network environment. For example, one or more emulation devices may perform one or more or all functions while fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices in the communication network. One or more emulation devices may perform one or more or all functions while temporarily implemented / deployed as part of a wired and / or wireless communication network. The emulation device may be directly coupled to another device for the purpose of testing and / or performing tests using over-the-air wireless communication.

[0063] The one or more emulation devices may perform one or more functions, inclusive, while not being implemented / deployed as part of a wired and / or wireless communication network. For example, the emulation devices may be utilized in test scenarios in a test lab and / or in an undeployed (e.g., test) wired and / or wireless communication network to implement testing of one or more components. The one or more emulation devices may be test equipment. Direct RF coupling and / or wireless communication via RF circuitry (which may include, for example, one or more antennas) may be used by the emulation devices to transmit and / or receive data.

[0064] A rendering / presentation engine (e.g., a game engine such as Unity, Unreal Engine, etc.) may be used to compose and render a scene. The information used to compose and render a scene, commonly referred to as scene information, may be provided by a scene description. One scene description format is Khronos' glTF. glTF is a JSON-based file format that provides a mechanism to reference external binary and image files. glTF supports static media objects, including dynamic characteristics in some formats that use animation. MPEG has commonly incorporated glTF as a scene graph format and is extending glTF to support dynamic (timed) content such as 2D and 360-degree video, dynamic volumetric visual media, and audio.

[0065] For example, there is the development of the international standard ISO / IEC 23090-14, which leverages the glTF specification and defines support for dynamic (timed) media such as 2D video, volumetric visual media, audio, haptics, etc. Support for dynamic media is enabled using an extension mechanism in glTF. The MPEG-I Scene Description Group in WG3 has begun to integrate MPEG-defined media formats such as 2D video encoded using the H.264 / AVC (ISO / IEC 14496-10), HEVC (ISO / IEC 23008-2), or VVC (ISO / IEC 23090-3) codecs.

[0066] In addition, methods are being investigated to provide support for volumetric visual media as defined in the ISO / IEC 23090-5 and ISO / IEC 23090-9 standards, and glTF via extensions. An Exploratory Experiment (EE) has been initiated to support MPEG immersive codecs in MPEG scene descriptions. The EE is to devise design principles and workflows for immersive media coded content using MPEG technologies such as Video-Based Point Cloud Compression (V-PCC), Geometry-Based Point Cloud Compression (G-PCC), and MPEG Immersive Media (MIV). In the latest draft of ISO / IEC 23090-14, efforts are being made to specify how to play volumetric visual media that is part of a glTF scene. Currently, two different approaches are being investigated as technologies under consideration to support Volumetric Visual Video-Based Coding (V3C) content. However, these approaches are limited and do not accommodate different types of V3C coded content.

[0067] Content encoded using the V3C codec is composed of several V3C components. These components contain some video coding information (e.g. geometry, occupancy, and attributes) and an atlas metadata stream that allows reconstruction of volumetric media frames using the video coding information. The frames for the video coding components follow a structure described by the atlas patch information. Each atlas frame can be subdivided into atlas tiles. An atlas tile can contain multiple patches. Each patch can contain additional information and each patch can be of a different type. For example, in the case of PROJECTED type, each patch can provide Point Local Reconstruction (PLR) information at either block level or patch level, and each attribute video stream is of only one attribute type.

[0068] A glTF extension that supports and enables the use of the YCbCr texture format in MPEG-I scene descriptions. The extension adds to the properties of YCbCr textures. A process is described that allows appropriate shader implementations or native GPU implementations to use the information in the extension. The extension adapts the YCbCr texture descriptions used in modern graphics APIs such as Vulkan.

[0069] A glTF extension to support timed 3D video in MPEG-I scene descriptions coded using V3C (ISO / IEC 23090-5) techniques. Such timed 3D video can be V-PCC (as enumerated in ISO / IEC 23090-5) and MIV (ISO / IEC 23090-12).

[0070] A semantic mechanism is introduced, by which the decoded V3C components are represented in the MPEG-I Scene Description. Each V3C component is represented as a JSON literal under the glTF extensions. Some of the components have specific properties that are further described in the respective components, namely V3C Attributes Component and V3C Atlas Component. Furthermore, a processing method is described for the presentation engine that utilizes the information stored under the glTF extensions.

[0071] A buffer format for the decoded atlas data provides the relevant information to reconstruct a representation of 3D video from the decoded 2D video stream. The buffer formats can be of different types, each type providing relevant application-specific information. In particular for MIV, the use of common atlas data to include view parameters is also expressed as a property in the glTF extension. Buffer formats that store the view parameters are exchanged with the presentation engine, thereby allowing different views to be reconstructed.

[0072] FIG. 2 illustrates an exemplary MPEG scene description architecture 200. There are two main entities in the MPEG-I scene description architecture 200: a media access function (MAF) 210 and a presentation engine (PE) 250. A fundamental consideration of the reference architecture 200 is to separate the functionality of the MAF 210 from the rendering and presentation of the PE 250. The MAF 210 is responsible for requesting, fetching, decoding, and post-processing the media data required by the PE 250 to render the various media objects that are part of the scene. The MAF 210 is expected to perform the presentation of the media data in an appropriate buffer format according to a scene description document 270 so that the media data is readable by the PE 250. The scene description document 270 is loaded by the PE 250, and the scene description document 270 identifies the associated buffers for each media object in the scene from their format.

[0073] The MAF 210 communicates with the cloud 240 to request media. These media requests can occur in either direction. The MAF 210 communicates with the local storage 260 to gain access to media. These media accesses can occur in either direction. The MAF 210 configures a media pipeline for rendering and uses a set of buffers 220 to do so. The buffer management 230 is configured to operate between the PE 250 and the MAF 210 to provide synchronized media for rendering. The PE 250 receives media in a synchronized format from the MAF 210 via the buffers 220 controlled by the buffer management 230 to render the scene.

[0074] FIG. 3 illustrates an example pipeline design 300 for processing V-PCC content. The example pipeline design 300 may be used in identifying an appropriate media pipeline and associated format for each media type to be supported. Different potential media pipelines may be configured to process media differently depending on the particular use case, and therefore each media pipeline may have a different output buffer format. One of the potential pipeline designs for processing V3C / V-PCC media is shown in FIG. 3 (Pipeline #2A 310.2A). In this pipeline 310.2a, the decoding and post-processing steps for V3C / V-PCC coded content provide processed data in well-defined buffers to the PE 250 for 3D reconstruction. Dedicated shader programs 320 may be utilized by the PE 250 to perform 3D reconstruction when using the workflow of Pipeline #2A 310.2a. The shader programs may be loaded into GPU hardware resources, resulting in efficient, real-time playback of volumetric visual media content.

[0075] In Figure 3A, pipeline #1 310.1 is illustrated. Pipeline #1 310.1 includes a single track 330 that feeds into a demultiplexer 332. Pipeline #1 310.1 then includes a series of HEVC decoders 334 and metadata 336. Pipeline #1 310.1 continues with a series of processes 338 that feed into a 3D reconstruction 340. Pipeline #1 310.1 is then buffered (350) and provides a signal to the PE 250.

[0076] 3 illustrates pipeline #2a 310.2a. Pipeline #2a 310.2a includes a geometry track 362, a texture track 364, an occupancy track 366, an atlas track 368, and static metadata 370. These tracks are in place of the single track 330 that feeds the demultiplexer 332 in pipeline #1 310.1. Pipeline #2a 310.2a then includes a series of HEVC decoders 372 and metadata 374. These series of HEVC decoders 372 and metadata 374 may be similar to the series of HEVC decoders 334 and metadata 336. Instead of performing the reconstruction within pipeline #2a 310.2a as in pipeline #1 310.1 in 3D reconstruction 320, pipeline #2a 310.2a is instead buffered within a series of buffers 378 (described herein) and 3D reconstruction 340 occurs within PE 250 as described above.

[0077] 3, pipeline #2b includes a single track 380 that is buffered 382 and fed to the PE 250. Demultiplexing / decoding / processing 384 may be performed on the signal within the PE 250.

[0078] To support the use of such workflows, a buffer format is defined for each V3C / V-PCC content. Furthermore, processing steps may perform all operations to represent data in a well-defined buffer format. Therefore, the scene description document must provide information about the buffers.

[0079] To support timed data access, the buffer element in ISO / IEC DIS 12113:2021 is extended to provide the functionality of a circular buffer. The extension is named MPEG_buffer_circular and may be included as part of the "buffer" structure. Buffers that provide access to timed data may include the MPEG_buffer_circular extension.

[0080] If the MPEG_buffer_circular extension is present in a buffer element, the buffer element property uri may not be present and the buffer element property byteLength may indicate the maximum possible size of the buffer. When present, the MPEG_buffer_circular extension may be included as an extension of the buffer object defined in ISO / IEC DIS 12113:2021.

[0081] The accessors specified in ISO / IEC DIS 12113:2021 define the type and layout of data stored in a buffer that is viewed through a bufferView object. When timed media is read from a buffer, the data in the buffer can change dynamically over time.

[0082] Timed accessor extensions allow access to dynamically changing data used in a scene. Timed accessors are extensions to regular glTF accessors to indicate that the underlying data buffer is dynamic. Timed accessors may have two buffer views, one inherited from the containing accessor and one inherited in the MPEG_accessor_timed extension. The former may be used to reference the timed media data, and the latter may be used to point to a dynamic buffer header, which may or may not be present. If both bufferViews are present, they may point to the same circular buffer. Accessors with the MPEG_accessor_timed extension may only point to buffers that contain the MPEG_buffer_circular extension, as described herein.

[0083] The accessor.bufferView field in the accessor with the MPEG_accessor_timed extension, as well as the timed accessor information header field, are applied to the data of each frame in the circular buffer. The timed accessor extension is identified by the MPEG_accessor_timed element. If the MPEG_accessor_timed extension is present, it may be included as an extension of the accessor object defined in ISO / IEC DIS 12113:2021. This extension, identified by the MPEG_texture_video element, provides the possibility to link texture objects defined in ISO / IEC DIS 12113:2021 to video sources. The MPEG_texture_video extension provides a reference to a timed accessor, i.e. an accessor with the MPEG_accessor_timed extension, so that decoded texture frames can be made available. If the MPEG_texture_video extension is present, it may be included as an extension of the texture element defined in ISO / IEC DIS 12113:2021. When the MPEG_texture_video extension is not supported, standard glTF textures may be used as a fallback.

[0084] The MPEG_texture_video extension contains an accessor property that provides a reference to an accessor by specifying the index of a particular accessor object in an accessor array that describes the buffer to which the decoded timed texture can be made available. The MPEG_texture_video extension also provides information about the format of the video texture through a format property. The accessor type, componentType, and count properties depend on the width, height, and format properties.

[0085] A media pipeline is a chain of media processing components for processing media. A media pipeline is specific to the media content. The patch packing block size is a unit that represents the 2D arrangement of patches within an atlas.

[0086] Multiple atlases may be included in a V3C bitstream. A V3C bitstream may be demultiplexed into one or more atlas bitstreams and their associated video bitstreams when multiple atlases are present. The video bitstream for each atlas may include occupancy, geometry, and attribute components. In the V3C parameter set, vps_atlas_count_minus1+1 indicates the total number of supported atlases in the current bitstream. The value of vps_atlas_count_minus1 may be in the range of 0 to 63, inclusive. vps_atlas_id specifies the ID of the atlas. The value of vcp_atlas_id may be in the range of 0 to 63, inclusive.

[0087] The V3C bitstream is multiplexed into one or more atlas sub-bitstreams, if multiple atlases are present, and into the associated video sub-bitstream. Each atlas video sub-bitstream may contain video coded occupancy, geometry, and attribute components. The atlas sub-bitstreams are decoded using an atlas decoder (a hypothetical reference atlas decoder is defined in Annex E of ISO / IEC 23090-5).

[0088] The atlas decoder parses the atlas sub-bitstream to decode the atlas NAL units and the atlas tile header. An atlas frame may be subdivided into tiles. Each atlas tile is independently decodable, facilitating implementations to perform efficient processing such as parallel processing. Each tile may contain a different number of patches. A patch stores information such as mapping information from 2D space to 3D space, patch type, etc. After decoding a patch present in an atlas tile, a mapping between blocks and patches is created, which assigns each block that overlaps with a patch with a patch index to the atlas tile in which the patch is contained (see clause 9.2.6 of ISO / IEC 23090-5). Blocks are indexed per patch per tile.

[0089] It is expected that in a V3C sequence, the number of total patches as well as the number of tiles may change over time. Therefore, it is not possible to provide deterministic information related to a patch with respect to the tile it belongs to. Rather, the tile-level patch information is transformed into atlas-frame-level patch information. Since the atlas frame is constant, i.e., the size of the atlas frame does not change in a V3C sequence.

[0090] The V3C specification provides a method to convert tile-level patch information to atlas-level patch information (ISO / IEC 23090-5, section 9.2.7). The operations defined in ISO / IEC 23090-5, section 9.2.7 convert the patch information (e.g., 2D location of the patch, patch dimensions, etc.) to be relative to the atlas frame instead of to a particular tile.

[0091] For example, section 9.2.7.2 of ISO / IEC 23090-5 provides information on how to convert a tile-level block-to-patch map to an atlas-level block-to-patch map. Such a mapping creates a list of all patches contained in the atlas frame. Every block is assigned a patch index with reference to the atlas frame (some blocks may not overlap with any patch; such blocks are indexed with -1). This operation removes the notion of tiles in the blockToPatch mapping structure.

[0092] To eliminate the notion of tiles in the decoded patch information, the operations defined in ISO / IEC 23090-5, section 9.2.7.3 are invoked. These operations transform patch information that is defined with reference to the tile that the patch is in to a global reference with respect to the atlas frame. These operations transform and provide atlas level patch information, such as information that is common to all patches, as well as application specific patch information.

[0093] The decoded atlas buffer contains two main structures: a block-to-patch map and a patch list. The block-to-patch map maps each block in the atlas frame to a patch index. If a block is not covered by any patch, the patch index value assigned to that block is -1. The patch list contains data for each patch contained in the patch list. A patch contains two sets of information: common patch information data and application-specific patch information data. Depending on the type of patch that needs to be stored in the atlas buffer, e.g., point local reconstruction, EOM point, etc., there may be additional information.

[0094] Common patch information can be applied throughout the patch, including (see ISO / IEC 23090-5, chapter 9.2.7.3.2) the following information: patch type, patch 2D size, patch position, patch 3D offset, patch 3D depth range, patch projection ID, patch orientation index, level of detail scaling factor, raw points, and number of EOM patches. Storing common patches in order can be beneficial.

[0095] Application-specific atlas-level patch information provides further details for each patch type (see ISO / IEC 23090-5, section 9.2.7.3.3). For example, when the patch type is PROJECTED, additional information may be required to perform point local reconstruction (PLR) (ISO / IEC 23090-5, section 8.4.7.9). Similarly, additional details may be provided for other patch types such as EOM and RAW.

[0096] MPEG Immersive Video is an extension of the V3C specification. MIV describes the concept of a view and the mapping of this concept to patches in the V3C component. A view represents a field of view of a volumetric frame, which can be represented by one 2D frame providing the geometry and one 2D frame per attribute. The coded atlas contains information describing the patches in the atlas frame. The coded atlas frame contains view information for each patch so that 3D points can be represented according to the reference frame of any view. For each patch, a different view is signaled that identifies which view the patch originates from. For each view, there are several parameters:

[0097] The application-specific atlas-level patch information is further extended to add a view index for each patch in the MIV content.

[0098] Video frames provided by the decoder may require additional processing steps before being input to the reconstruction process. The process described in Annex B of ISO / IEC 23090-5 is invoked for decoded video components associated with the same atlas ID. Different V3C video coding components are processed and undergo different processes. The V3C video coding components are converted to a nominal format. The nominal format collectively refers to the nominal bit depth, resolution, chroma format, and composition time index to which the decoded video may be converted. Each video sub-bitstream and each region of the packed video sub-bitstream is associated with a nominal bit depth, which is the target bit depth at which all operations for reconstruction are expected to be performed. In ISO / IEC 23090-14, the nominal format is expected to be a 4:4:4 chroma subsampling format. The processes may include map extraction, geometric map synthesis, attribute map synthesis, resolution conversion, and chroma upsampling. For details of the processes, see Annex B of ISO / IEC 23090-5.

[0099] Packed video is a flexible syntax that allows multiple components of a V3C bitstream to be packed into the same video frame. Multiple V3C video components are included in the same V3C unit, i.e., V3C_PVD. The V3C_PVD may store video components, such as some, multiple, or all of the video components of an atlas (e.g., geometry, occupancy, and attribute video). The V3C syntax may describe frame packing logic. After decoding of the packed video, the frame packing logic may be used by a V3C decoder to unpack the various components into separate individual video component streams.

[0100] When vuh_unit_type is equal to V3C_PVD, a decoding process corresponding to the packed video component may be invoked. Information about the packed video configuration is provided in the packing information syntax. The packing information syntax may provide information for unpacking the packed video stream.

[0101] 4 illustrates an exemplary V3C profile 400. In general, a V3C profile follows a structured and flexible definition to allow for unambiguous identification of two distinct conformance points: conformance point A 410 and conformance point B 420. It may be mandatory to indicate conformance point A 410, while conformance point B 420 may be optional. The decoding capabilities of the V3C profile 400 are defined by a codec group 450 and a toolset 460, which typically reside within the VPS or may be retrieved via external means.

[0102] The first conformance point, conformance point A 410, covers the decoded video sub-bitstream and the atlas sub-bitstream. The first conformance point 410 covers the derived block-to-patch map information. However, the first conformance point 410 does not cover the reconstruction process. Any decoder conforming to V3C profile 400 at conformance point A410 (identified by syntax elements ptl_profile_codec_group_idc and ptl_profile_toolset_idc) at a particular level 440 (identified by the value of syntax element ptl_level_idc) of a particular tier 430 (identified by the value of syntax element ptl_tier_flag) may be able to decode all V3C bitstreams or a collection of V3C sub-bitstreams according to Section A.2 of ISO / IEC 23090-5, which provides details on how profile 480, toolset 460 and tier 430 information is used to decode V3C media.

[0103] The second conformance point, conformance point B 420, covers the reconstruction process. The third V3C profile component, reconstruction 470, describes conformance point B 420 and specifies pre-reconstruction, reconstruction, post-reconstruction, and adaptation tools that are supported or recommended to achieve conformance for 3D reconstruction.

[0104] glTF does not natively support video textures. The MPEG_texture_video extension mentioned above provides the ability to support timed textures. However, when the color space of the information stored in a video texture is YUV / YCbCr, appropriate sampler information may be required.

[0105] Typically, a texture item in the texture array has two properties: sampler and source. The source property provides information about the texture, and the sampler property provides information related to the sampling operation. A typical syntax for a texture item in the texture array is shown below. The sampler property for a particular texture object in the texture array points to a sampler object in the sampler array, and the source property for the texture object points to an image object in the image array.

[0106] A typical syntax for a texture item in a texture array is as follows:

[0107] [Table 1]

[0108] Texture sources in glTF typically support JPEG (JFIF) and PNG image formats. However, decoding the source image files is left to the implementation; the decoding process is not expressed in glTF.

[0109] In many cases, data for source images may be stored as interleaved data or planar data. There is no support for expressing source image data alignment in glTF.

[0110] The atlas information is specified as the syntax of the MPEG_primitive_v3c extension in CDAM1, as illustrated in Table 1.

[0111] [Table 2]

[0112] The _MPEG_V3C_AD object illustrated in Table 2 may have the following structure:

[0113] [Table 3]

[0114] Currently, CDAM1 of ISO / IEC 23090-14 specifies one version (version 1) of the buffer format for storing Common Atlas data, as shown in Table 3.

[0115] [Table 4]

[0116] A generic and scalable design is useful for supporting V3C encoded content in an MPEG-I scene description. The design supports different types of patches and patch information. The design may also configure information required to reconstruct volumetric visual content (e.g., point clouds). The design may also provide information about the reconstruction process and V3C components. A complete design may support representing atlas metadata information within an MPEG-I scene description document such that the atlas information may be used in combination with video components decoded by PE 250 for reconstruction operations.

[0117] The described system and method presents a generic design for supporting V3C / V-PCC content, including semantics and binary representation for decoded V3C / V-PCC media data in a buffer format that can be used by PE 250 for reconstruction and rendering. The system and method may require definition in two stages: data representation and semantics description. Both the data representation and their semantics may be defined in a new glTF extension, referred to herein as MPEG_V3C extension.

[0118] The proposed system and method distributes the processing of V3C content between the MAF and the PE 250, where the MAF 210 performs, in addition to the nominal format conversion, the operations required for conformance point A 410 as defined in the ISO / IEC 23090-5 specification, and the PE 250 performs the 3D reconstruction process associated with conformance point B 420 (see Annex A of ISO / IEC 23090-5).

[0119] The data representation of the different components of V3C / V-PCC is important to distinguish the formats of the different components.

[0120] The reconstruction process of volumetric media utilizes information from tiles of an atlas frame. Therefore, conversion of tile level patch information to atlas level patch information is necessary as presented herein. The atlas level patch information is stored in a binary buffer. Specific information, such as common atlas level patch information and application atlas level patch information as well as other related information presented herein, can be retrieved by defining different accessors to the buffer. Each accessor points to the same binary buffer with a different bufferView. Each bufferView may have a different binary offset and a different binary length to access each sub-block in the buffer data. Each sub-block in the binary buffer stores a finite length scalar value.

[0121] For example, the blockToPatch information block may store an array of scalar values ​​with length equal to the total number of patch packing blocks, which can be determined from formulas (1) and (2) in ISO / IEC 23090-5 and are described below for convenience. AtlasBlockToPatchMapWidth=(asps_frame_width+offset) / PatchPackingBlockSize Formula (1) AtlasBlockToPatchMapHeight=(asps_frame_height+offset) / PatchPackingBlockSize Formula (2)

[0122] In general, asps_frame_height and asps_frame_width may be read by the atlas decoder during the decoding process. Thus, MAF 210 may calculate the total number of patch packing blocks in the atlas frame. The total number of patches in the atlas frame is retrieved using the TotalNumberOfPatches sub-block in the atlas data buffer frame. TotalNumberOfPatches may also count the total number of different patch types stored in the atlas frame. Similarly, the binary sub-block storing common patch parameters may store 16 defined parameters for each patch in an array of scalar values. Thus, the total number of parameters stored in this array is equal to the total number of patches in the atlas frame multiplied by 16. Thus, PE 250 may implement logic for an atlas data reader that can effectively read data from the atlas binary buffer (using information retrieved from multiple accessors). These patch parameters are stored in the following order:

[0123] [Table 5]

[0124] Figure 5 illustrates an example atlas data representation 500 for patch types PROJECTED and PLR enabled at the patch level. Figure 5 presents an example of atlas level patch information data 510 for an atlas frame. Each sub-block in the atlas data can be accessed through the following dedicated accessors: Atlas data representation 500 includes BlockToPatch information 510, total number of patches 530, common patch parameters 540, AtlasPlrdLevel 550, AtlasPlrdPresentFlag 560, and AtlasPlrdMode 570.

[0125] FIG. 6 illustrates multiple accessors that access different sub-blocks within the atlas binary data. FIG. 6 presents an example having multiple accessors, each providing access to a different sub-block within the atlas data. FIG. 6 illustrates this information starting with the atlas data representation 500 of FIG. 5. Each sub-block of the atlas data is provided with an accessor. For example, BlockToPatch information 510 is identified by accessor 620, total number of patches 530 is identified by accessor 630, common patch parameters 540 are identified by accessor 640, AtlasPlrdLevel 550 is identified by accessor 650, AtlasPlrdPresentFlag 560 is identified by accessor 660, and AtlasPlrdMode 570 is identified by accessor 670.

[0126] Figure 7 illustrates an atlas data representation with patch type PROJECTED and PLR information available at the block level. Figure 7 presents an example of atlas level patch information data for an atlas frame where PLR ​​information is available at the block level rather than at the patch level as illustrated in Figure 6. Figure 7 illustrates this information starting from atlas data representation 500 of Figure 5. Each sub-block of the atlas data is provided with patch type PROJECTED and PLR information. For example, BlockToPatch information 510 is identified with patch type PROJECTED and PLR information 720, total number of patches 530 is identified with patch type PROJECTED and PLR information 730, common patch parameters 540 are identified with patch type PROJECTED and PLR information 740, AtlasPlrdLevel 550 is identified with patch type PROJECTED and PLR information 750, AtlasPlrdPresentFlag 560 is identified with patch type PROJECTED and PLR information 760, and AtlasPlrdMode 570 is identified with patch type PROJECTED and PLR information 770.

[0127] Thus, depending on the type of patch and the appropriate flags that determine the information available to the patch, the structure of the atlas data in the binary buffer may differ. However, the means of accessing the data from the atlas binary buffer remain the same, i.e., using accessors where each accessor provides access to an array of scalar values ​​whose length is determined by the total number of parameters. If the binary data is timed data (i.e., the media is dynamic), timed accessors may be used. The timed accessors described above provide a means to access a binary buffer that is dynamic and whose properties may change over time. In the case of partial access, only a few tiles of the V3C content are needed, and the MAF 210 is responsible for requesting the relevant / target tile. However, the concept of tiles does not exist at all in the atlas data stored in the buffer, so the same design principles of atlas data access are applicable when only partial content is retrieved. If the content is MIV, a different number of views and per-view parameters are provided in the atlas frame. The view parameters for each view are stored in the atlas buffer along with other atlas information.

[0128] The set of accessors (of FIG. 6) provides a means to access patch-specific information for each of the patches in the atlas frame while referencing a single binary buffer element. The single buffer may be referenced by a set of bufferViews, and each bufferView may have its own glTF accessor element. The use of the accessors allows the PE 250 to access all of the information associated with the patches contained in the decoded atlas frame.

[0129] 8 illustrates a representation 800 of a sub-block and a method of reading the atlas data. The sub-block may contain V3C atlas data corresponding to the semantics in some embodiments of n described below, and the sub-block may contain V3C atlas data corresponding to the semantics in Table 16. In an alternative approach, the use of accessor objects is minimized (particularly for application-specific information), thus compressing the associated data.

[0130] The following description provides details on how the atlas data is compressed. The sub-block of data that stores the BlockToPatchMap information may follow a defined structure. The data stores a patch index for each block in the atlas frame. Thus, the properties of the accessor for the blocktoPatchMap information are: components: unsignedInt or float, type: scalar, and count: total number of blocks in the atlas frame.

[0131] The sub-block representing the number of patches may store the total number of patches as well as the total number of application-specific patches. The accessor properties are: componentType: unsignedInt, type: scalar, count: total number of patches + total number of (projected patches + eom patches + RAW patches).

[0132] The sub-block storing common patch parameter information may store per-patch patch parameters in a list. The per-patch common patch parameters may be ordered. The data may be stored either as a scalar entity or as a vec2 entity. If the scalar type is selected, the data may follow an implicit ascending order of patch index as well as patch parameters. If the vec2 type is selected, the data may be stored as presented in Table 1. The properties of the timed accessor header information are: componentType:unsignedInt, type:scalar / vec2, count:total number of patches * The number is 16.

[0133] 8, representation 800 includes a number of accessors (collectively MPEG accessor timers 810) including accessor 1 810.1, accessor 2 810.2, accessor 3 810.3, and accessor 4 810.4, although any number of accessors may be used. A number of MPEG accessor timers (collectively MPEG accessor timers 820) including an MPEG accessor timer 820.1 for accessor 1 810.1, an MPEG accessor timer 820.2 for accessor 2 810.2, an MPEG accessor timer 820.3 for accessor 3 810.3, and an MPEG accessor timer 820.4 for accessor 4 810.4. Then multiple buffers (collectively buffers 830) are fed, including buffer 1 830.1 from MPEG accessor timer 820.1, buffer 2 830.2 from MPEG accessor timer 820.2, buffer 3 830.3 from MPEG accessor timer 820.3, and buffer 4 830.4 from MPEG accessor timer 820.4. Other buffers, such as buffer 835, may be fed directly by the accessor 810, MPEG accessor timer 820, for example.

[0134] Buffer 840 (or multiple buffers) may be fed by buffer 830 and buffer 835. Representation 800 includes multiple timed accessor header information blocks 850, including timed accessor header information block 850.1 fed by buffer 830.1, timed accessor header information block 850.2 fed by buffer 830.2, timed accessor header information block 850.3 fed by buffer 830.3, and timed accessor header information block 850.4 fed by buffer 830.4 and buffer 840. The timed accessor header information 850 may then be fed to a descriptor 860. For example, timed accessor header information 850.1 is fed to descriptor 860.1, timed accessor header information 850.2 is fed to descriptor 860.2, timed accessor header information 850.3 is fed to descriptor 860.3, and timed accessor header information 850.4 is fed to descriptor 860.4.

[0135] The buffer 835 may provide information to the atlas frame data 845, which may include block-to-patch map information 870, NOPs 875, common patch parameters 880, and application-specific patch parameters 890. The descriptor 860 may be fed back to the atlas frame data 845, which includes the block-to-patch map information 870, NOPs 875, common patch parameters 880, and application-specific patch parameters 890.

[0136] [Table 6]

[0137] Depending on the type of patch, there may be additional application-specific parameter information. To store such information, the data storage is divided into patch types, which are also visually represented in FIG.

[0138] If the patch type is PROJECTED, the point local reconstructions may be stored in the atlas buffer. There may be at least two sub-blocks that store the PLR ​​data, depending on whether the PLR ​​information is available at the block level or the patch level. Information related to which PLR level each projected patch corresponds to is provided in the sub-block that stores the data, as represented in Table 6. The corresponding accessor properties for the sub-blocks are: componentType:unsignedInt, type:vec2, count:number of PROJECTED patches.

[0139] [Table 7]

[0140] Depending on the PLR ​​level included in the atlas frame, there may be either information about the PLR ​​of each block in the patch or PLR information for the patch (i.e., blocks in a patch may have the same PLR ​​information). If the PLR ​​level for the patch is identified as block level, then the sub-blocks may have block level PLR information stored as represented in Table 7. The corresponding accessor properties for the sub-blocks are: componentType:unsignedInt, type:vec3, count:

[0141]

number

[0142] [Table 8]

[0143] If the PLR ​​level for the patch is identified as patch-level, the sub-block may be the stored patch-level PLR information as represented in Table 8. The corresponding accessor properties for the sub-block are componentType: unsignedInt, type: vec3, and count: number of patch-level PLR patches.

[0144] [Table 9]

[0145] For the EOM patch type, information related to the EOM patch may be stored as shown in Table 9. The corresponding accessor properties for the subblock are: componentType: unsignedInt, type: vec3, and count:

[0146]

number

[0147] [Table 10]

[0148] If the atlas frame stores MIV related data, there may be a list of camera views corresponding to each patch. The MIV views are stored as shown in Table 10 or Table 11. The corresponding accessor properties for the subblocks are: componentType: unsignedInt, type: vec2, and count: number of views (NOV). * (27+NOV).

[0149] [Table 11]

[0150] The corresponding accessor properties for the subblock are: componentType: unsignedInt, type: scalar, and count: 1 + number of views (NOV). * (27+NOV).

[0151] [Table 12]

[0152] MIV also extends application specific patch parameter information (see ISO / IEC23090-12, chapter 9.2.7.3.3). Thus there can be additional sub-blocks that store information. Each patch corresponds to a view index. Thus the patch list stores the view index per patch. The corresponding accessor properties for the sub-blocks are: componentType:unsignedInt, type:vec2 / scalar, count:total number of patches.

[0153] If the patch type is PROJECTED, additional parameters are provided as shown in Table 12. The MIV projection parameters for each patch may be stored in a certain order. The corresponding accessor properties for the subblock are: componentType: unsignedInt, type: vec2 / scalar, and count: total number of PROJECTED patches. * It could be 5.

[0154] [Table 13]

[0155] V3C video coded components such as occupancy, geometry, and attributes are typically decoded using a conventional video decoder such as H.264, HEVC, or VVC. The decoded frames from each V3C component may also undergo additional post-processing processes (see Appendix B ISO / IEC 23090-5). The final decoded frames for the V3C component are made available to the PE 250 as video textures.

[0156] In the context of a scene description, a decoded video texture is signaled as a texture object in a texture array. A video texture object is a texture object that includes an MPEG_texture_video extension that extends the capabilities of texture elements to support video textures. A video texture provides access to the decoded video texture as well as associated information. The video texture can be processed to represent the video texture in a nominal format that can be used by the PE 250 for 3D reconstruction by performing post-processing operations on the decoded V3C video texture. The format of the decoded video texture can be a 4:4:4 chroma subsampling format. In one configuration, the nominal chroma format is 4:4:4, since it is important to have a representation that matches every sample in the decoded V3C image to a reconstructed 3D point. Thus, the video texture can be in YCbCr format or RGB format. The decision of which color format is used for the video texture (i.e., RGB or YCbCr) is application / implementation specific.

[0157] A new scene-level extension, MPEG_V3C, signals information related to each V3C component of V3C media objects present in a scene and may describe textures and buffers that may hold decoded video frames and atlas data for these media objects. The extension enumerates an array of V3C media objects. Each item in the array contains information about the components of the corresponding V3C media object, signaled using four main properties: V3C_OCCUPANCY, which represents information related to the occupancy component; V3C_GEOMETRY, which represents information related to the geometry component; V3C_ATTRIBUTE, which represents information related to the attribute component; and V3C_ATLAS, which represents information related to the atlas component of the V3C content. A V3C content may contain two or more atlases, and each atlas may be associated with its own set of geometry, occupancy, and attribute components.

[0158] In a glTF file, the "MPEG_V3C" extension may be added to the "extensionRequired" and "extensionUsed" top-level glTF properties, which may include the following:

[0159] [Table 14]

[0160] The V3C syntax may be a top-level extension. A node in a scene with an MPEG_V3C extension may reference an index in an array defined in an MPEG_V3C element in the scene-level extension. An example pseudo-glTF file utilizing the MPEG_V3C extension is provided below, providing a syntax for representing V3C content. This syntax for representing V3C content includes:

[0161] [Table 15-1]

[0162] [Table 15-2]

[0163] [Table 15-3]

[0164] If asps_map_count_minus1>0, this means that there are two or more maps in the atlas frame. In such a case, multiple PROJECTED patch entries can be described in the V3C_ATLAS property of MPEG_V3C.

[0165] [Table 16]

[0166] V3C syntax may be a mesh-level extension. According to the glTF specification, accessors referenced by attributes in mesh.primitives store the specified attribute data for the vertices of the mesh. Accessors to each attribute in mesh.primitive may store per-vertex data and therefore have the same value for the count property of each attribute's accessor. For example, the following pseudo-glTF example describes a triangle mesh with a POSITION attribute and a NORMAL attribute. Each attribute points to an accessor. The accessor provides information on how to read the data for the attribute. Three vertices are needed to describe a triangle. The accessor at index 1 stores POSITION data for the three vertices. The accessor at index 2 stores NORMAL data for the three vertices. The count property for both the accessor at index 1 and the accessor at index 2 is the same.

[0167] [Table 17-1]

[0168] [Table 17-2]

[0169] [Table 17-3]

[0170] The glTF specification allows new attribute types to be defined to store application-specific data about a mesh. However, such new attributes may follow the data representation rules of glTF, i.e., the accessors for the application-specific attribute must define per-vertex data and each attribute's accessors must have the same count value. To represent a V3C as a mesh in glTF, the syntax elements used to describe the V3C may not break the core data representation concepts of a glTF mesh. For example, if one defines a mesh.primitive.attribute that does not store per-vertex data for all vertices of the mesh, such a glTF file should be considered invalid.

[0171] Since V3C is a different kind of compression scheme for 3D objects, i.e. projection-based, it can be considered as a candidate to be used to compress 3D objects, i.e. point clouds in glTF meshes. An extension to mesh.primitives can be provided. This extension design is similar to the design used in defining "KHR_draco_mesh_compression". In the case of V3C, the extension can describe the decoded but not reconstructed V3C content. Once the reconstruction is performed, the 3D object is in its true format, i.e. a point cloud. The representation of point clouds using glTF.mesh fits the mesh definition and follows a common construct to define new attributes. These new attributes can be V3C specific attributes such as "REFLECTANCE", "MATERIAL_ID", etc.

[0172] Thus, the syntax of the V3C object may be provided as an extension to the mesh in the scene description file. An extension to the mesh object is provided. The extension is a feature specific to the object coded in the V3C compression scheme. The presentation client is equipped with a graphic processing unit (GPU). A glTF loader may parse the glTF file. If the parser supports reconstruction of V3C objects and accepts the "MPEG_V3C" extension, the glTF loader in the PE 250 may first process the "MPEG_V3C" extension for the mesh. The loader may load the decoded V3C data represented by the extension. The decoded V3C data provided by the properties specified by the MPEG_V3C extension is loaded onto the GPU memory. An implementation (e.g., a shader implementation) is performed on the decoded V3C data to generate the final reconstructed 3D object. The logic of the 3D reconstruction is driven using V3C information such as atlas, geometry, occupancy, etc. The 3D object may be further textured using texture information with different V3C attributes. The reconstructed 3D data may be stored in a separate buffer on the GPU memory. The primitive's "POSITION" attribute corresponds to an accessor that stores the position of the vertices of the reconstructed and reconstructed 3D object, the primitive's "NORMAL" attribute corresponds to an accessor that stores the normal value of the vertices of the reconstructed and reconstructed 3D object in the "ATTR_TEXTURE" attribute of the V3C content, and the primitive's "COLOR" attribute corresponds to an accessor that stores the color value of the vertices of the reconstructed and reconstructed 3D object in the "ATTR_TEXTURE" attribute of the V3C content. After the 3D reconstruction, the 3D object is in its true format, i.e., a point cloud. Thus, mesh.primitive.attribute represents the 3D object with attribute accessors with per-vertex data. Thus, the count property of each mesh.primitive.attribute may be the same.There may be other attributes such as "_REFLECTANCE", "_MATERIAL_ID", "_TRANSPARENCY", etc. that correspond to different V3C attribute types defined for each vertex of the 3D object.

[0173] The mesh.primitive.extension is named "MPEG_V3C" and refers to the decoded data of a V3C object. Each V3C component is provided by a property in the MPEG_V3C extension.

[0174] The syntax of the extension is presented below.

[0175] [Table 18]

[0176] The extensions are expressed at the mesh level, so that a node that references a mesh with the extension "MPEG_V3C" can introduce the object into the scene graph for rendering.

[0177] MIV is a special case of V3C representation as a mesh in glTF2.0. The glTF loader may store in an accessor the viewing space bounds referenced by the mesh's "POSITION" attribute. The viewing space bounds are conveyed by the MIV bitstream. Textures in the MIV content are passed directly to the renderer based on different camera views in the viewing space. Information about the camera views is stored in the atlas property of the MPEG_V3C extension.

[0178] There can be multiple atlases multiplexed in a V3C bitstream. Each atlas can have its corresponding atlas data and video coded components such as geometry, attributes and occupancy. There are therefore two approaches that can be followed to define support for multiple atlases for MPEG_V3C extensions. The two approaches are detailed below.

[0179] One approach includes an outer-inner array for each V3C component. To describe support for multiple atlases, each property in the MPEG_V3C extension provides an outer array. The outer array length may be equal to the number of atlases for the V3C object. Components with maps such as geometry, occupancy, and attributes further reference an array, the inner array, where each array item references a specific map. Properties in the inner array correspond to component-specific data references, e.g., for video coding data, the index of the corresponding video texture is referenced. Similarly, for atlas data in V3C_ATLAS, different accessors reference respective buffers that store respective atlas data for atlas items in the array. Each item in the outer array with index i of a component may have a corresponding component for the same atlas in other properties at index i. Index i may be an atlas ID. For example, an item with index i in V3C_ATLAS may correspond to atlas data for the atlas at index i. Corresponding video coded components for the same atlas, such as attributes, are referenced by the item at index i in the V3C_ATTRIBUTE property. In addition to what is explicitly stated, an additional property named "atlasID" is introduced for each atlas in the MPEG_V3C extension. atlasID is an array of integer values. Each integer value references the vpc_atlas_id shown above for each atlas in the V3C bitstream.

[0180] Using this approach, a syntax description is presented below.

[0181] [Table 19]

[0182] Another exemplary approach includes an array of atlases. Another approach to describe support for multiple atlases is to define a new property under the "MPEG_V3C" extension named "atlas". An "atlas" is an array of components that correspond to atlases, as shown in Table 13. The length of the atlas array may be equal to the number of atlases for a V3C object. The properties for the objects in the atlas array describe the atlas data components and corresponding video coding components, such as attributes, occupancy, and geometry for the V3C object.

[0183] [Table 20]

[0184] The objects in the "atlas" array correspond to the components of one atlas frame of the V3C object. The objects may have an identifier named "atlas_ID" that stores the same integer value as vps_atlas_id. Using this approach, the syntax description is presented below.

[0185] [Table 21]

[0186] The semantics for the MPEG_V3C extensions at the scene level are described in Table 14 below.

[0187] [Table 22]

[0188] The semantics for the atlas data are explained in Tables 15 and 16.

[0189] [Table 23]

[0190] [Table 24]

[0191] In contrast to the V3C_atlas syntax defined herein, in another embodiment, alternative syntaxes for V3C_atlas are possible. Depending on the type of patch (i.e., PROJECTED, EOM, or RAW), additional information may be provided. For example, an atlas frame consists of PROJECTED patches, and each projected patch may have point local reconstruction (PLR) information. Whether PLR information for a patch is available at the block level or at the patch level is provided through "PLRLevel", and the corresponding PLR data is provided in "BlockPLRD" and / or "PatchPLRD". "BlockSize" corresponds to the value of the patch packing blocking size.

[0192] The following example defines the properties of a V3C Atlas component for V3C content that contains a PROJECTED patch and an EOM patch. The data representation of the syntax elements is presented in Tables 5, 6, 7, 8, 9, 10, and 11.

[0193] [Table 25]

[0194] If asps_map_count_minus1>0, there are two or more maps in the atlas frame. In such a case, multiple PROJECTED patch entries can be described in the V3C_ATLAS property of MPEG_V3C. Therefore, the PLR ​​information for each patch can be marked with a map index, i.e., PLRLevel_N, where N is the map index.

[0195] [Table 26]

[0196] Different embodiments of semantics for V3C_atlas are also contemplated. These semantics correspond to the syntax defined in this specification. The V3C atlas information may be provided with a smaller number of accessor units. Valid accessor and component types for each property of the V3C atlas frame are defined in Table 17.

[0197] [Table 27]

[0198] Two different approaches are described below for storing application-specific information for a given patch in the atlas frame.

[0199] First, application-specific information is stored separately from the common patch parameters. To store "application-specific" data in the atlas frame for different patch types, as described above, a JSON object is introduced in the "MPEG_V3C" extension that references accessors that store in the application-specific data for each patch type. A "V3C_application_specific_info" property is introduced in the MPEG_V3C extension and can have the following child properties, as shown in Table 18:

[0200] [Table 28]

[0201] The "V3C_application_specific_info" property is defined as a property in the "MPEG_V3C" extension at the primitive level.

[0202] [Table 29]

[0203] The data representation of the application specific information for the different properties stores patch type specific information defined in the ISO / IEC 23090-5 specification. Table 19 provides the data representation for the parameters of the "PROJECTED" property in "V3C_application_specific_info".

[0204] [Table 30]

[0205] Table 20 provides the data representation for the parameters of the “PROJECTED” property in “V3C_application_specific_info” for MIV ISO / IEC23090-12 content.

[0206] [Table 31]

[0207] Table 21 provides the data representation of the parameters of the "EOM" property in "V3C_application_specific_info".

[0208] [Table 32]

[0209] Table 22 provides the data representation for the parameters of the "RAW" property in "V3C_application_specific_info".

[0210] [Table 33]

[0211] Second, application-specific information is stored together with the common patch information. Currently, CDAM1 of ISO / IEC23090-14 specifies one version (version 1) of the buffer format for storing common atlas data. This has limitations because it does not include syntax for storing PLR information, EOM patch type information, RAW patch type information, and MIV-specific information. Therefore, a list of new versions of the atlas buffer format is introduced in Table 23.

[0212] [Table 34]

[0213] Below are a series of buffer format definitions in Tables 24, 25, 26, 27, and 28 for different atlas buffer format versions as specified in Table 23.

[0214] [Table 35-1]

[0215] [Table 35-2]

[0216] The calculation of blockCount is specified in ISO / IEC 23090-5, clause 8.4.7.9, function BlockCnt(xSize, ySize). The arguments of the function are the tile patch dimensions, i.e., 2d_size_x and 2d_size_y.

[0217] [Table 36]

[0218] [Table 37]

[0219] [Table 38-1]

[0220] [Table 38-2]

[0221] [Table 39]

[0222] Common Atlas Data V3C semantics can be used in MPEG-I scene descriptions. Common Atlas Data is applicable across the CVS. Therefore, additional properties are defined under the MPEG_V3C extensions, as shown in Table 29.

[0223] [Table 40]

[0224] Some of the common atlas information that is common to atlases in CVS is specified in ISO / IEC 23090-12. A syntax for MIV extensions to the common atlas data is specified. The syntax provides information about the list of view parameters that can be used during the rendering process (Annex H.1 of ISO / IEC 23090-12). A property named "MIV_view_parameters" is specified in the _MPEG_V3C_CAD object as shown in Table 30. In any future iterations, there may be more properties defined for the MPEG_V3C_CAD property.

[0225] [Table 41]

[0226] The syntax of MPEG_V3C_CAD is presented below.

[0227] [Table 42]

[0228] Table 31 provides the data representation for the parameters of the "MIV_view_parameters" property in "MPEG_V3C_CAD". These parameters are defined in Chapter 9.7.5.1 of ISO / IEC 23090-12.

[0229] [Table 43-1]

[0230] [Table 43-2]

[0231] [Table 43-3]

[0232] Support for V3C packed video components may be included in the MPEG-I scene description. Since information about the packed video layout is available in the V3C syntax, MAF 210 may unpack the packed video stream after decoding the packed video stream. After unpacking the decoded packed video (Annex B.4 in ISO / IEC 23090-5), MAF 210 may indicate the different decoded and unpacked V3C components, such as V3C_GEOMETRY, V3C_ATTRIBUTE, and V3C_OCCUPANCY, in their associated properties in the "MPEG_V3C" extension. In this approach to processing packed video in V3C, no new syntax needs to be introduced in the MPEG_V3C extension because all V3C components are defined separately.

[0233] The occupancy and geometry components are associated with their respective texture objects in the texture array through an index property, whose semantics are described in Table 32.

[0234] [Table 44]

[0235] For an attribute component, there is an identification of the type of attribute, in addition to the index of the associated texture object. Thus, the properties associated with an attribute component are as described in Table 33.

[0236] [Table 45]

[0237] The key identifiers represented by the value of the type property relate to V3C attribute types as shown in Table 33. The types in Table 34 correspond to the defined attribute types in Table 14 of clause 8.4.4.5 of ISO / IEC 23090-5.

[0238] [Table 46]

[0239] A top-level extension is introduced herein. The extension is called "MPEG_V3C". In one embodiment, a top-level glTF extension "MPEG_OBJECTS" may be provided. "MPEG_OBJECTS" stores a list of objects. Each object item that describes an object may be extended. Below is an example of a pseudo-glTF file that introduces the MPEG_OBJECTS extension:

[0240] [Table 47]

[0241] Each item in the object array may be extended to include a component description for the media. For example, the MPEG_V3C extension may describe the V3C components of a media item as shown in this specification. In this approach, multiple objects coded using MPEG codec technology may be referred to as object items. To attach MPEG_OBJECTS items to nodes, a node-level extension is introduced to reference objects defined in the top-level MPEG_OBJECTS.object array. Using such a mechanism is flexible in supporting other types of MPEG coded content, such as G-PCC. As described in this paragraph, any content coded using MPEG technology may be referred to as an object item in the MPEG_OBJECTS.object array. The content may be coded using any MPEG technology, such as V-PCC, G-PCC, etc.

[0242] The semantics for objects in the top-level MPEG_OBJECT extension are described in Table 35.

[0243] [Table 48]

[0244] An object-level glTF extension, such as "MPEG_V3C", may contain data corresponding to different components of a V3C object. The syntax of the different components for a V3C object is presented herein. The semantics for objects in the node-level MPEG_OBJECTS extension are described in Table 36.

[0245] [Table 49]

[0246] In a glTF file, the "MPEG_OBJECTS" and "MPEG_V3C" extensions can be added to the "extensionRequired" and "extensionUsed" top-level glTF properties.

[0247] The semantics for the MPEG_V3C extension functions at the node level are described in Table 37 below.

[0248] [Table 50]

[0249] In addition to the MPEG_V3C extension, certain content-specific information may assist in performing conformance point validation as described herein. Media is accessed through the MPEG_media extension defined in ISO / IEC 23090-14. Content-specific information may be provided through a JSON object in the extraParams property of an alternative in the alternative array of a V3C media object signaled in the MPEG_media extension. The syntax of parameters for performing conformance point validation for V3C content is described below.

[0250] [Table 51]

[0251] The semantics for the different parameters required to perform fitness point validation are explained in Table 38.

[0252] [Table 52]

[0253] The information in the extraParams property is useful for bitstream decoding and 3D reconstruction. Possible values ​​for each of the parameters in Table 38 are provided in Annex A of ISO / IEC 23090-5.

[0254] Modern graphics APIs, such as the Vulkan API, provide a sampler structure associated with each texture object so that sampling operations for the YCbCr color space can be natively supported on modern GPUs. A sampler level extension is described for natively sampling video textures in parallel processing devices such as GPUs. The format of the video texture, such as "VK_FORMAT_G8_B8_R8_3PLANE_444_UNORM", can be provided in the MPEG_video_texture.format property. The syntax is provided below:

[0255] [Table 53]

[0256] Texture objects in a texture array may use samplers with an "MPEG_YUV" sampler extension to provide information to the PE 250 about how to sample a video texture when the sample data in the texture is in a chroma format such as YCbCr. In a glTF file, the "MPEG_YUV" extension may be added to the "extensionRequired" and "extensionUsed" top-level glTF properties.

[0257] Table 39 provides a description of the properties defined in the MPEG_YUV sampler extension.

[0258] [Table 54]

[0259] Table 40 provides the definition of the top-level object of the MPEG_texture_video extension.

[0260] [Table 55]

[0261] Figure 9 illustrates a method 900 for a media workflow for V3C content as described herein. The following is a description of procedures that the MAF may use to configure a media pipeline for decoding and processing V3C media content and for the PE 250 to use in reconstructing buffer data. The procedures described in this section provide information related to the workflow for the media pipeline (pipeline #2A310.2a) as presented in Figure 9.

[0262] The PE 250 loads a scene graph (e.g., from a glTF file). The glTF includes nodes that represent single or multiple media content (e.g., volumetric visual media content).

[0263] The media content (e.g., volumetric visual media) is encoded. The media is encoded using a V3C / V-PCC encoder.

[0264] The encoded media content is referenced via a media item in the MPEG_media extension. The extraParams property provides additional information about the profile, layer, and codec group.

[0265] The MAF 210 starts the media pipeline for V3C media objects. The MAF 210 performs three operations: Extract 910, Decode 920, and Process 930. The V3C / V-PCC bitstreams may be multiplexed. The MAF 210 performs a demultiplexing operation to extract the different V3C components encapsulated in the bitstream. The decoding of the V3C components is performed using a typical video decoder such as HEVC, VVC, etc. The atlas track is decoded using an atlas decoder. The processing of the decoded V3C components converts the decoded frames to a nominal format, which is typically 4:4:4. Additional processes such as map extraction, chroma subsampling, etc. may be performed.

[0266] The processed V3C components are provided as video textures, which are referenced through glTF.texture, where the glTF.texture array has an MPEG_texture_video extension, and the color format of the video texture can be YUV / YCbCR. The texture object references a sampler with an "MPEG_YUV" extension that provides sampling information for the decoded video texture in the nominal format, and the YUV to RGB conversion can be done natively in the GPU using the sampler information.

[0267] The processed atlas data is provided through a binary buffer 940, as described herein, which is accessed by means of an accessor, which may have an MPEG_accessor_timed extension to allow for timed data.

[0268] The PE 250 uses binary buffers as well as video textures for the 3D reconstruction 950. The PE 250 can also use information related to the reconstruction profile provided in the extraParams property.

[0269] The MAF 210 requests media from the server based on the viewer's view of the scene.

[0270] Although the features and elements are described above in certain combinations, one skilled in the art can understand that each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein can be implemented in a computer program, software, or firmware embodied in a computer-readable medium for execution by a computer or processor. Examples of computer-readable media include electronic signals (transmitted over wired or wireless connections) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random-access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. 1. A method for using an extension for a Video Texture format in a Moving Pictures Expert Group (MPEG) scene description, the method comprising: receiving information about an MPEG-I scene, the information including a chroma texture; processing said chroma texture using an extension to provide information to a presentation engine (PE) for sampling a video texture; rendering the scene using the received MPEG-I scene information and the processed chroma textures.

2. The method of claim 1 , wherein the chroma texture comprises YCbCr.

3. The method of claim 1 , wherein the enhancement is configured for a chroma texture.

4. The method of claim 3 , wherein the enhancement step adapts the chroma texture.

5. The method of claim 3 , wherein the chroma textures are from a modern graphics API.

6. The method of claim 1 , wherein the scene is processed using a shader implementation.

7. The method of claim 1 , wherein the scene is processed using a GPU implementation.

8. The method of claim 1 , wherein the chroma textures are referenced through a glTF texture array.

9. The method of claim 8 , wherein the glTF texture array has an MPEG_texture_video extension.

10. 10. The method of claim 9, wherein the chroma texture references a sampler having an "MPEG_YUV" extension that provides sampling information for a decoded video texture in a nominal format.

11. The method of claim 1 , further comprising converting YUV to RGB in a GPU using the sampler information.

12. The method of claim 11 , wherein the conversion occurs natively.

13. 2. The method of claim 1, further comprising: requesting, via a Media Access Function (MAF), the information about the MPEG-I scene from a server based on one or more views by a viewer of the MPEG-I scene.

14. 1. A wireless transmit / receive unit (WTRU) configured to use an extension to support and enable use of a video texture format in an MPEG-I scene description, the WTRU comprising: A transceiver; a processor operably coupled to the transceiver, the transceiver and processor; receiving information about the MPEG-I scene, the information including a chroma texture; processing said chroma texture using an extension to provide information to a presentation engine (PE) for sampling a video texture; The WTRU is operable to render the scene using the received MPEG-I scene information and the processed chroma textures.

15. The WTRU of claim 14 , wherein the enhancement is configured for chroma textures.

16. The WTRU of claim 14 , wherein the scene is processed using a shader implementation.

17. The WTRU of claim 14 , wherein the scene is processed using a GPU implementation.

18. The WTRU of claim 14 , wherein the chroma textures are referenced through a glTF texture array.

19. The WTRU of claim 18 , wherein the glTF texture array has an MPEG_texture_video extension.

20. 20. The WTRU of claim 19, wherein the texture refers to a sampler having an "MPEG_YUV" extension that provides sampling information for a decoded video texture in a nominal format.

Images