Region of interest and / or viewport dependent delivery of V3C data using RTP

Abstract

Region of interest and / or viewport dependent delivery of V3C data may be performed using RTP. The RTP / RTCP signaling may support spatial region-based and / or viewport-based partial access to the V3C content. The SDP parameters may signal static 3D regions in the immersive media content. The RTCP FB message type may carry a 3D region of interest request during an RTP media transport session. The SDP parameter may indicate the RTCP-based capability to request the desired 3D region during the capability negotiation. The RTP header extension type may carry the transmitted 3D area information during RTP transmission of the immersive media. The SDP parameter may indicate an RTP-based capability to signal the transmitted 3D area information during capability negotiation. The SDP parameter may indicate an RTP-based capability to signal updated 3D area information during capability negotiation. The RTCP FB message type may carry viewport information during the RTP media transport session. The SDP parameter may indicate an RTCP-based capability to signal viewport information during capability negotiation.

Description

Cross-references

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 415,893, filed on October 13, 2022, and U.S. Provisional Application No. 63 / 539,958, filed on September 22, 2023, the contents of which are incorporated herein by reference. Background Art

[0002] Video coding systems may be used to compress digital video signals, for example, to reduce the storage and / or transmission bandwidth required for such signals. Video coding systems may include, for example, block-based, wavelet-based, and / or object-based systems. Summary of the invention

[0003] This article describes systems, methods, and tools for performing viewport-and / or region-of-interest-dependent delivery of visual volumetric video-based coding (V3C) data, for example, using a real-time transport protocol (RTP). The RTP / Real-time Control Protocol (RTCP) signaling mechanism can be used to support spatial region-based and / or viewport-based partial access to V3C content. Session Description Protocol (SDP) parameters can be used to signal static 3D regions present in immersive media content. RTCP feedback (FB) message types can carry a desired 3D region of interest request, for example, during RTP media transmission of a session. RTCP FB messages can be signaled from a receiver to a sender. SDP parameters can indicate an RTCP-based ability to request a desired 3D region, for example, during capability negotiation. RTP header extension types can carry the 3D region information sent during RTP transmission of immersive media. RTP header extension types can be signaled from a sender to a receiver.

[0004] Capability negotiation may be performed between a sender and a receiver of V3C content. SDP parameters may indicate, for example, an RTP-based capability to signal the sent 3D region information during capability negotiation. SDP parameters may indicate, for example, an RTP-based capability to signal updated 3D region information during capability negotiation. RTCP FB message types may carry desired and / or requested viewport information, for example, during RTP media transmission of a session. RTCP FB message types may be signaled from a receiver to a sender. SDP parameters may indicate, for example, an RTCP-based capability to signal desired and / or requested viewport information during capability negotiation.

[0005] A device may be configured to receive a set of session description protocol (SDP) parameters indicating the presence of one or more 3D regions associated with immersive content. The device may send a real-time control protocol (RTCP) feedback (FB) message indicating a three-dimensional (3D) region of interest and / or a 3D viewport of interest. The device may receive one or more 3D regions of visual volumetric video-based coding (V3C) content associated with the 3D region of interest and / or the 3D viewport of interest.

[0006] The device may perform capability negotiation (eg, using SDP). The set of SDP parameters may be received during capability negotiation.

[0007] The received set of 3D region information may be received using a real-time protocol (RTP) header extension, and the use of the RTP header extension may be signaled in an SDP message.

[0008] In an example, the 3D region of interest may be a static 3D region and / or an arbitrary 3D region. The RTCP FB message may include an indication of a region ID associated with the 3D region of interest, an indication of a position associated with the 3D region of interest, and / or a size associated with the 3D region of interest.

[0009] The RTCP FB message may indicate whether extrinsic camera parameters are present in the RTCP message, whether intrinsic camera parameters are present in the RTCP message, and / or whether the horizontal FOV associated with the 3D viewport of interest and the vertical FOV associated with the 3D viewport of interest are equal.

[0010] The systems, methods, and tools described herein may relate to decoders. In some examples, the systems, methods, and tools described herein may relate to encoders. In some examples, the systems, methods, and tools described herein may relate to signals (e.g., signals from encoders and / or received by decoders). A computer-readable medium may include instructions for causing one or more processors to perform the methods described herein. A computer program product may include instructions that, when executed by one or more processors, cause one or more processors to perform the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1A is a system diagram illustrating an example communication system in which one or more disclosed embodiments may be implemented.

[0012] Figure 1B is an example of a method that can be used according to an embodiment of the present invention. Figure 1AA system diagram of an example wireless transmit / receive unit (WTRU) for use within the illustrated communication system.

[0013] Figure 1C is an example of a method that can be used according to an embodiment of the present invention. Figure 1A System diagram of an example Radio Access Network (RAN) and an example Core Network (CN) for use within the illustrated communication system.

[0014] Figure 1D is an example of a method that can be used according to an embodiment of the present invention. Figure 1A System diagram of yet another example RAN and yet another example CN for use within the illustrated communication system.

[0015] Figure 2 An example video encoder is illustrated.

[0016] Figure 3 An example video decoder is illustrated.

[0017] Figure 4 Examples of systems in which various aspects and examples may be implemented are illustrated.

[0018] Figure 5 An example spatial subdivision of a point cloud is shown. DETAILED DESCRIPTION

[0019] A more detailed understanding may be obtained from the following description given by way of example in conjunction with the accompanying drawings.

[0020] Figure 1A 1 is a diagram illustrating an example communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content such as voice, data, video, messaging, broadcast, etc. to multiple wireless users. The communication system 100 may enable multiple wireless users to access such content through sharing of system resources (including wireless bandwidth). For example, the communication system 100 may employ one or more channel access methods such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single carrier FDMA (SC-FDMA), zero tail unique word DFT spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block filtered OFDM, filter bank multi-carrier (FBMC), etc.

[0021] like Figure 1AAs shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106 / 115, public switched telephone network (PSTN) 108, Internet 110 and other networks 112, but it should be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d (any of which may be referred to as a “station” and / or “STA”) may be configured to send and / or receive wireless signals and may include user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular phone, a personal digital assistant (PDA), a smart phone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (IoT) device, a watch or other wearable device, a head-mounted display (HMD), a vehicle, a drone, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in an industrial and / or automated process chain environment), a consumer electronic device, a device operating on a commercial and / or industrial wireless network, etc. Any of the WTRUs 102a, 102b, 102c, and 102d may be interchangeably referred to as a UE.

[0022] The communication system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device that is configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN 106 / 115, the Internet 110, and / or other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node B, an eNode B, a Home Node B, a Home eNode B, a gNB, an NR Node B, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each depicted as a single element, it will be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0023] The base station 114a may be part of the RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), a relay node, etc. The base station 114a and / or the base station 114b may be configured to send and / or receive wireless signals on one or more carrier frequencies (which may be referred to as cells (not shown)). These frequencies may be in a licensed spectrum, an unlicensed spectrum, or a combination of licensed and unlicensed spectrums. A cell may provide coverage of wireless services to a specific geographic area, which may be relatively fixed or may change over time. The cell may be further divided into cell sectors. For example, a cell associated with the base station 114a may be divided into three sectors. Therefore, in one embodiment, the base station 114a may include three transceivers, i.e., one transceiver for each sector of the cell. In an embodiment, the base station 114a may employ multiple-input multiple-output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to send and / or receive signals in a desired spatial direction.

[0024] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0025] More specifically, as noted above, the communication system 100 may be a multiple access system and may employ one or more channel access schemes such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, the base station 114a in the RAN 104 / 113 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may use Wideband CDMA (WCDMA) to establish the air interface 115 / 116 / 117. WCDMA may include communication protocols such as High Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High Speed ​​UL Packet Access (HSUPA).

[0026] In an embodiment, base station 114a and WTRUs 102a, 102b, 102c may implement a radio technology such as evolved UMTS terrestrial radio access (E-UTRA), which may use Long Term Evolution (LTE) and / or Long Term Evolution-Advanced (LTE-A) and / or LTE-A Pro to establish an air interface 116.

[0027] In an embodiment, base station 114a and WTRUs 102a, 102b, 102c may implement a radio technology such as NR radio access, which may use New Radio (NR) to establish an air interface 116.

[0028] In an embodiment, base station 114a and WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, base station 114a and WTRUs 102a, 102b, 102c may implement LTE radio access and NR radio access together using, for example, the dual connectivity (DC) principle. Accordingly, the air interface used by WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions to / from multiple types of base stations (e.g., eNBs and gNBs).

[0029] In other embodiments, base station 114a and WTRUs 102a, 102b, 102c may implement radio technologies such as IEEE 802.11 (i.e., Wi-Fi), IEEE 802.16 (i.e., WiMAX), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rate for GSM Evolution (EDGE), GSM EDGE (GERAN), etc.

[0030] Figure 1AThe base station 114b in the may be, for example, a wireless router, a Home NodeB, a Home eNodeB, or an access point, and may utilize any suitable RAT to facilitate wireless connectivity in a local area, such as a business location, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a road, and the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology (such as IEEE 802.11) to establish a wireless local area network (WLAN). In an embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology (such as IEEE 802.15) to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish a picocell or a femtocell. As Figure 1A As shown, base station 114b may have a direct connection to Internet 110. Therefore, base station 114b may not need to access Internet 110 via CN 106 / 115.

[0031] The RAN 104 / 113 may be in communication with the CN 106 / 115, which may be any type of network configured to provide voice, data, applications, and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have different quality of service (QoS) requirements, such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. The CN 106 / 115 may provide call control, billing services, mobile location-based services, prepaid calling, Internet connectivity, video distribution, etc., and / or perform advanced security functions, such as user authentication. Although not described in detail in the text and video sections of the RAN 104 / 113, the CN 106 / 115 may be configured to provide voice, data, applications, and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have different quality of service (QoS) requirements, such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. The CN 106 / 115 may provide call control, billing services, mobile location-based services, prepaid calling, Internet connectivity, video distribution, etc., and / or perform advanced security functions, such as user authentication. Figure 1A Although not shown in the figure, it will be appreciated that the RAN 104 / 113 and / or the CN 106 / 115 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 104 / 113 or a different RAT. For example, in addition to being connected to the RAN 104 / 113, which may utilize NR radio technology, the CN 106 / 115 may also be in communication with another RAN (not shown) that employs GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0032] The CN 106 / 115 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network that provides plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the Transmission Control Protocol (TCP), the User Datagram Protocol (UDP), and / or the Internet Protocol (IP) in the TCP / IP Internet protocol suite. The networks 112 may include wired communication networks and / or wireless communication networks owned and / or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs, which may employ the same RAT as the RAN 104 / 113 or a different RAT.

[0033] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communication system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). Figure 1A The illustrated WTRU 102c may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[0034] Figure 1B is a system diagram illustrating an example WTRU 102. Figure 1B As shown, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138. It will be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0035] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal decoding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. Although Figure 1B The processor 118 and the transceiver 120 are depicted as separate components, but it is understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0036] The send / receive element 122 may be configured to send a signal to a base station (e.g., base station 114a) or receive a signal from the base station by an air interface 116. For example, in one embodiment, the send / receive element 122 may be an antenna configured to send and / or receive an RF signal. In an embodiment, the send / receive element 122 may be a transmitter / detector configured to send and / or receive, for example, IR, UV or visible light signals. In another embodiment, the send / receive element 122 may be configured to send and / or receive both RF signals and optical signals. It should be understood that the send / receive element 122 may be configured to send and / or receive any combination of wireless signals.

[0037] Although the transmit / receive element 122 Figure 1B 1 as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0038] The transceiver 120 may be configured to modulate signals to be transmitted by the transmit / receive element 122 and to demodulate signals received by the transmit / receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. For example, the transceiver 120 may include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0039] The processor 118 of the WTRU 102 may be coupled to and may receive user input data from a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from and store data in any type of suitable memory, such as a non-removable memory 130 and / or a removable memory 132. The non-removable memory 130 may include a random access memory (RAM), a read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from and store data in a memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[0040] The processor 118 may receive power from the power source 134, and may be configured to distribute and / or control power to other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0041] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to or in lieu of the information from the GPS chipset 136, the WTRU 102 may receive location information from a base station (e.g., base stations 114a, 114b) over the air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information by any suitable location-determination method while remaining consistent with an embodiment.

[0042] The processor 118 may also be coupled to other peripherals 138, which may include one or more software modules and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, module, a frequency modulation (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, etc. Peripheral device 138 may include one or more sensors, which may be one or more of the following: a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0043] The WTRU 102 may include a full-duplex radio for which transmission and reception of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous. The full-duplex radio may include an interference management unit for reducing and / or substantially eliminating self-interference via signal processing performed via hardware (e.g., a choke) or via a processor (e.g., a separate processor (not shown) or via the processor 118). In an embodiment, the WRTU 102 may include a half-duplex radio for which transmission and reception of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) or downlink (e.g., for reception)) may be concurrent and / or simultaneous.

[0044] Figure 1C 1 is a system diagram illustrating the RAN 104 and the CN 106 in accordance with an embodiment. As noted above, the RAN 104 may employ an E-UTRA radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 104 may also be in communication with the CN 106.

[0045] The RAN 104 may include evolved Node-Bs 160a, 160b, 160c, though it will be appreciated that the RAN 104 may include any number of evolved Node-Bs while remaining consistent with an embodiment. The evolved Node-Bs 160a, 160b, 160c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the evolved Node-Bs 160a, 160b, 160c may implement MIMO technology. Thus, the evolved Node-B 160a, for example, may use multiple antennas to transmit wireless signals to and / or receive wireless signals from the WTRU 102a.

[0046] Each of the evolved Node Bs 160a, 160b, 160c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in UL and / or DL, etc. Figure 1C As shown, the eNode-Bs 160a, 160b, 160c may communicate with one another via an X2 interface.

[0047] Figure 1C The illustrated CN 106 may include a mobility management entity (MME) 162, a serving gateway (SGW) 164, and a packet data network (PDN) gateway (or PGW) 166. While each of the foregoing elements is depicted as being part of the CN 106, it should be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0048] The MME 162 may be connected to each of the evolved Node-Bs 162a, 162b, 162c in the RAN 104 via an S1 interface and may serve as a control node. For example, the MME 162 may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, bearer activation / deactivation, selecting a particular serving gateway during an initial attach of the WTRUs 102a, 102b, 102c, etc. The MME 162 may provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as GSM and / or WCDMA.

[0049] The SGW 164 may be connected to each of the evolved Node-Bs 160a, 160b, 160c in the RAN 104 via an S1 interface. The SGW 164 may generally route and forward user data packets to / from the WTRUs 102a, 102b, 102c. The SGW 164 may perform other functions, such as anchoring the user plane during inter-evolved Node-B handovers, triggering paging when DL data is available for the WTRUs 102a, 102b, 102c, managing and storing the context of the WTRUs 102a, 102b, 102c, and the like.

[0050] The SGW 164 may be connected to the PGW 166, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices.

[0051] The CN 106 may facilitate communications with other networks. For example, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices. For example, the CN 106 may include, or may be in communication with, an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) that acts as an interface between the CN 106 and the PSTN 108. In addition, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to other networks 112, which may include other wired networks and / or wireless networks owned and / or operated by other service providers.

[0052] Although the WTRU Figures 1A to 1D Although described as wireless terminals, it is contemplated that in certain representative embodiments, such terminals may (eg, temporarily or permanently) use a wired communications interface with a communications network.

[0053] In a representative embodiment, the other network 112 may be a WLAN.

[0054] A WLAN in infrastructure basic service set (BSS) mode may have an access point (AP) for the BSS and one or more stations (STA) associated with the AP. The AP may have access or an interface to a distribution system (DS) or another type of wired / wireless network that carries traffic to and / or carries traffic away from the BSS. Traffic originating from outside the BSS and leading to the STA may be reached by the AP and may be delivered to the STA. Traffic originating from the STA and leading to a destination outside the BSS may be transmitted to the AP to be delivered to the corresponding destination. Traffic between STAs within the BSS may be transmitted by the AP, for example, wherein the source STA may transmit traffic to the AP, and the AP may deliver traffic to the destination STA. Traffic between STAs within the BSS may be considered and / or referred to as point-to-point traffic. Point-to-point traffic may be transmitted between the source STA and the destination STA (e.g., directly between them) using a direct link establishment (DLS). In certain representative embodiments, the DLS may use 802.11e DLS or 802.11z tunnel DLS (TDLS). A WLAN using an independent BSS (IBSS) mode may not have an AP, and STAs within or using the IBSS (eg, all STAs in the STA) may communicate directly with each other. The IBSS communication mode may sometimes be referred to herein as an "ad hoc" communication mode.

[0055] When using the 802.11ac infrastructure operating mode or a similar operating mode, the AP may send beacons on a fixed channel (such as a primary channel). The primary channel may be a fixed width (e.g., a 20 MHz wide bandwidth) or a width dynamically set via signaling. The primary channel may be an operating channel of the BSS and may be used by the STA to establish a connection with the AP. In certain representative embodiments, carrier sense multiple access / collision avoidance (CSMA / CA) may be implemented, for example, in an 802.11 system. For CSMA / CA, a STA (e.g., each STA) (including the AP) may listen to the primary channel. In the case where the primary channel is listened / detected and / or determined to be busy by a specific STA, the specific STA may back off. One STA (e.g., only one station) may transmit at any given time in a given BSS.

[0056] High throughput (HT) STAs may communicate using a 40 MHz wide channel, for example, via a combination of a primary 20 MHz channel and adjacent or non-adjacent 20 MHz channels to form a 40 MHz wide channel.

[0057] Very high throughput (VHT) STA can support 20MHz, 40MHz, 80MHz and / or 160MHz wide channels. 40MHz channels and / or 80MHz channels can be formed by combining continuous 20MHz channels. 160MHz channels can be formed by combining 8 continuous 20MHz channels, or by combining two non-continuous 80MHz channels (this can be called 80+80 configuration). For 80+80 configuration, after channel coding, the data can pass through a segment parser that can divide the data into two streams. Each stream can be processed by inverse fast Fourier transform (IFFT) and time domain processing separately. These streams can be mapped to two 80MHz channels, and data can be sent by the transmitter STA. At the transmitter of the receiving STA, the above-mentioned operation for the 80+80 configuration can be reversed, and the combined data can be transmitted to the medium access control (MAC).

[0058] 802.11af and 802.11ah support operating modes below 1GHz. Channel operating bandwidths and carriers are reduced in 802.11af and 802.11ah relative to those used in 802.11n and 802.11ac. 802.11af supports 5MHz, 10MHz, and 20MHz bandwidths in the TV White Space (TVWS) spectrum, and 802.11ah supports 1MHz, 2MHz, 4MHz, 8MHz, and 16MHz bandwidths using non-TVWS spectrum. According to a representative embodiment, 802.11ah may support instrument type control / machine type communications, such as MTC devices in macro coverage areas. MTC devices may have certain capabilities, such as limited capabilities, including support for (e.g., only support for) certain bandwidths and / or limited bandwidths. MTC devices may include batteries with battery life above a threshold (e.g., to maintain very long battery life).

[0059] WLAN systems that can support multiple channels and channel bandwidths (such as 802.11n, 802.11ac, 802.11af, and 802.11ah) include channels that can be designated as primary channels. The primary channel may have a bandwidth equal to the maximum common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel may be set and / or limited by a STA (which supports the minimum bandwidth operating mode) from all STAs operating in the BSS. In the example of 802.11ah, for STAs (e.g., MTC-type devices) that support (e.g., only support) a 1MHz mode, the primary channel may be 1MHz wide, even if the AP and other STAs in the BSS support 2MHz, 4MHz, 8MHz, 16MHz, and / or other channel bandwidth operating modes. Carrier sensing and / or network allocation vector (NAV) settings may depend on the state of the primary channel. If the primary channel is busy, for example, because a STA (supporting only a 1MHz operating mode) is sending to the AP, the entire available frequency band may be considered busy even if most of the frequency bands remain idle and may be available.

[0060] In the United States, the available frequency band for 802.11ah is 902MHz to 928MHz. In South Korea, the available frequency band is 917.5MHz to 923.5MHz. In Japan, the available frequency band is 916.5MHz to 927.5MHz. The total bandwidth available for 802.11ah is 6MHz to 26MHz, depending on the country code.

[0061] Figure 1D 1 is a system diagram illustrating the RAN 113 and the CN 115 in accordance with an embodiment. As noted above, the RAN 113 may employ NR radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 113 may also be in communication with the CN 115.

[0062] The RAN 113 may include gNBs 180a, 180b, 180c, though it will be appreciated that the RAN 113 may include any number of gNBs while remaining consistent with an embodiment. The gNBs 180a, 180b, 180c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the gNBs 180a, 180b, 180c may implement MIMO technology. For example, the gNBs 180a, 180b may utilize beamforming to send signals to and / or receive signals from the gNBs 180a, 180b, 180c. Thus, the gNB 180a may, for example, use multiple antennas to send wireless signals to and / or receive wireless signals from the WTRU 102a. In an embodiment, the gNBs 180a, 180b, 180c may implement carrier aggregation technology. For example, gNB 180a may transmit multiple component carriers to WTRU 102a (not shown). A subset of these component carriers may be on unlicensed spectrum, while the remaining component carriers may be on licensed spectrum. In an embodiment, gNBs 180a, 180b, 180c may implement coordinated multi-point (CoMP) technology. For example, WTRU 102a may receive coordinated transmissions from gNB 180a and gNB 180b (and / or gNB 180c).

[0063] The WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using transmissions associated with a set of parameters that may be scalable. For example, OFDM symbol spacing and / or OFDM subcarrier spacing may vary for different transmissions, different cells, and / or different portions of the wireless transmission spectrum. The WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using subframes or transmit time intervals (TTIs) of varying or scalable lengths (e.g., containing different numbers of OFDM symbols and / or varying absolute time lengths over time).

[0064] The gNBs 180a, 180b, 180c may be configured to communicate with the WTRUs 102a, 102b, 102c in a standalone configuration and / or a non-standalone configuration. In a standalone configuration, the WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c while not accessing other RANs (e.g., such as the eNodeBs 160a, 160b, 160c). In a standalone configuration, the WTRUs 102a, 102b, 102c may use one or more of the gNBs 180a, 180b, 180c as mobility anchors. In a standalone configuration, the WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using signals in an unlicensed band. In a non-standalone configuration, the WTRUs 102a, 102b, 102c may communicate / connect with the gNB 180a, 180b, 180c while also communicating / connecting with another RAN, such as the eNode-B 160a, 160b, 160c. For example, the WTRUs 102a, 102b, 102c may implement the DC principle to communicate with one or more gNBs 180a, 180b, 180c and one or more eNode-Bs 160a, 160b, 160c substantially simultaneously. In a non-standalone configuration, the eNode-B 160a, 160b, 160c may act as a mobility anchor for the WTRUs 102a, 102b, 102c, and the gNB 180a, 180b, 180c may provide additional coverage and / or throughput for serving the WTRUs 102a, 102b, 102c.

[0065] Each of the gNBs 180a, 180b, 180c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in UL and / or DL, support of network slicing, dual connectivity, interworking between NR and E-UTRA, routing of user plane data towards a user plane function (UPF) 184a, 184b, routing of control plane information towards an access and mobility management function (AMF) 182a, 182b, etc. Figure 1D As shown, gNBs 180a, 180b, and 180c may communicate with each other via an Xn interface.

[0066] Figure 1DThe illustrated CN 115 may include at least one AMF 182a, 182b, at least one UPF 184a, 184b, at least one session management function (SMF) 183a, 183b, and possible data networks (DNs) 185a, 185b. Although each of the foregoing elements is depicted as part of the CN 115, it should be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0067] The AMF 182a, 182b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via the N2 interface and may act as a control node. For example, the AMF 182a, 182b may be responsible for authenticating users of the WTRU 102a, 102b, 102c, support of network slicing (e.g., handling of different PDU sessions with different requirements), selecting a specific SMF 183a, 183b, management of registration areas, termination of NAS signaling, mobility management, etc. The AMF 182a, 182b may use network slicing to customize CN support for the WTRU 102a, 102b, 102c based on the type of services utilized by the WTRU 102a, 102b, 102c. For example, different network slices may be established for different use cases, such as services relying on ultra-reliable low latency (URLLC) access, services relying on enhanced mobile broadband (eMBB) access, services for machine type communication (MTC) access, etc. The AMF 162 may provide a control plane function for switching between the RAN 113 and other RANs (not shown) that employ other radio technologies, such as LTE, LTE-A, LTE-A Pro, and / or non-3GPP access technologies, such as WiFi.

[0068] The SMF 183a, 183b may be connected to the AMF 182a, 182b in the CN 115 via the N11 interface. The SMF 183a, 183b may also be connected to the UPF 184a, 184b in the CN 115 via the N4 interface. The SMF 183a, 183b may select and control the UPF 184a, 184b, and configure the traffic routing through the UPF 184a, 184b. The SMF 183a, 183b may perform other functions, such as managing and allocating UE IP addresses, managing PDU sessions, controlling policy implementation and QoS, providing downlink data notification, etc. The PDU session type may be IP-based, non-IP-based, Ethernet-based, etc.

[0069] The UPF 184a, 184b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via the N3 interface, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices. The UPF 184, 184b may perform other functions, such as routing and forwarding packets, enforcing user plane policies, supporting multi-homed PDU sessions, handling user plane QoS, buffering downlink packets, providing mobility anchoring, etc.

[0070] The CN 115 may facilitate communications with other networks. For example, the CN 115 may include, or may communicate with, an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) that acts as an interface between the CN 115 and the PSTN 108. In addition, the CN 115 may provide the WTRUs 102a, 102b, 102c with access to other networks 112, which may include other wired networks and / or wireless networks owned and / or operated by other service providers. In one embodiment, the WTRUs 102a, 102b, 102c may be connected to local data networks (DNs) 185a, 185b via the UPF 184a, 184b via an N3 interface to the UPF 184a, 184b and an N6 interface between the UPF 184a, 184b and the DN 185a, 185b.

[0071] Given that Figures 1A to 1D as well as Figures 1A to 1D Corresponding to the description of the present invention, one or more or all of the functions described herein with reference to one or more of the following may be performed by one or more simulation devices (not shown): WTRU102a-d, base station 114a-b, evolved Node B 160a-c, MME 162, SGW 164, PGW 166, gNB 180a-c, AMF182a-b, UPF 184a-b, SMF 183a-b, DN 185a-b and / or any other device described herein. The simulation device may be one or more devices configured to emulate one or more or all of the functions described herein. For example, the simulation device may be used to test other devices and / or simulate network and / or WTRU functions.

[0072] The simulation device may be designed to implement one or more tests of other devices in a laboratory environment and / or in an operator network environment. For example, one or more simulation devices may perform one or more functions or all functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network in order to test other devices within the communication network. One or more simulation devices may perform one or more functions or all functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. The simulation device may be directly coupled to another device for testing purposes and / or may use over-the-air wireless communications to perform testing.

[0073] One or more simulation devices can perform one or more (including all) functions without being implemented / deployed as part of a wired and / or wireless communication network. For example, the simulation device can be used in a test scenario in a test laboratory and / or a non-deployed (e.g., testing) wired and / or wireless communication network to implement testing of one or more components. One or more simulation devices can be test equipment. Direct RF coupling and / or wireless communication via RF circuits (e.g., which may include one or more antennas) can be used by the simulation device to send and / or receive data.

[0074] The present application describes a number of aspects, including tools, features, examples, models, methods, etc. Many of these aspects are described in a particular manner, and at least in order to illustrate individual features, are usually described in a manner that may sound restrictive. However, this is to describe clearly and does not limit the application or scope of these aspects. In fact, all different aspects can be combined and interchanged to provide further aspects. In addition, these aspects can also be combined and interchanged with the aspects described in the earlier submission.

[0075] The aspects described and contemplated in this application can be implemented in many different forms. Figure 5 to Figure 2 1 may provide some examples, but other examples are also contemplated. Figure 5 to Figure 2 1 does not limit the breadth of the specific implementations. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to sending a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the methods.

[0076] In this application, the terms "reconstruction" and "decoding" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image", "picture" and "frame" may be used interchangeably.

[0077] Various methods are described herein, and each method includes one or more steps or actions for implementing the method. Unless the correct operation method requires a specific order of steps or actions, the order and / or purpose of specific steps and / or actions can be modified or combined. In addition, in various examples, terms such as "first", "second" and the like can be used to modify elements, parts, steps, operations, etc., such as "first decoding" and "second decoding". Unless specifically required, the use of such terms does not imply the ordering of the modification operation. Therefore, in this example, the first decoding does not need to be performed before the second decoding, and can, for example, occur before, during, or in an overlapping time period of the second decoding.

[0078] like Figure 2 and Figure 3 As shown, various methods and other aspects described in this application can be used to modify modules (e.g., decoding modules) of video encoder 200 and decoder 300. In addition, the subject matter disclosed herein can be applied to, for example, any type, format, or version of video decoding (whether described in a standard or described in a recommendation), whether pre-existing or developed in the future, as well as extensions of any such standards and recommendations. Unless otherwise indicated or technically excluded, the aspects described in this application can be used alone or in combination.

[0079] Various numerical values ​​are used in the examples described in this application, such as bit positions, bit numbers, byte numbers, field numbers, parameter values, parameter value ranges, group numbers, group positions, 3D region IDs, coordinates, etc. These and other specific values ​​are for the purpose of describing the examples, and the aspects are not limited to these specific values.

[0080] Figure 2 2 is a schematic diagram illustrating an example video encoder. Variations of the example encoder 200 are contemplated, but the following describes encoder 200 for clarity without describing all contemplated variations.

[0081] Before being encoded, the video sequence may undergo a pre-encoding process (201), for example, applying a color transform to an input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0082] In encoder 200, a picture is encoded by encoder elements as described below. The picture to be encoded is partitioned (202) and processed in units such as coding units (CUs). Each unit is encoded, for example, using intra mode or inter mode. When a unit is encoded in intra mode, the unit performs intra prediction (260). In inter mode, motion estimation (275) and compensation (270) are performed. The encoder decides (205) which of intra mode or inter mode to use to encode the unit, and indicates the intra / inter decision by, for example, a prediction mode flag. The prediction residual is calculated, for example, by subtracting (210) the predicted block from the original image block.

[0083] The prediction residual is then transformed (225) and quantized (230). The quantized transform coefficients, along with motion vectors and other syntax elements (e.g., picture partition information), are entropy coded (245) to output a bitstream. The encoder may skip the transform and apply quantization directly to the untransformed residual signal. The encoder may bypass both the transform and quantization, i.e., directly code the residual without applying the transform or quantization process.

[0084] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The image block is reconstructed by combining the decoded prediction residual and the predicted block (255). An in-loop filter (265) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) / ALF (adaptive loop filter) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (280).

[0085] Figure 3 is a schematic diagram illustrating an example of a video decoder. In the example decoder 300, a bitstream is decoded by decoder elements as described below. The video decoder 300 generally performs the same Figure 2 The decoding process is the reverse of the encoding process described in . The encoder 200 also typically performs video decoding as part of encoding the video data.

[0086] Specifically, the input of the decoder includes a video bitstream, which may be generated by the video encoder 200. First, the bitstream is entropy decoded (330) to obtain transform coefficients, prediction modes, motion vectors, and other decoded information. Picture partition information indicates how the picture is partitioned. Therefore, the decoder can divide (335) the picture according to the decoded picture partition information. The transform coefficients are dequantized (340) and inverse transformed (350) to decode the prediction residual. The image block is reconstructed by combining (355) the decoded prediction residual and the predicted block. The predicted block can be obtained (370) from intra-frame prediction (360) or motion compensated prediction (i.e., inter-frame prediction) (375). An in-loop filter (365) is applied to the reconstructed image. The filtered image is stored at a reference picture buffer (380). In some examples, for a given picture, the content of the reference picture buffer 380 on the decoder 300 side may be the same as the content of the reference picture buffer 280 on the encoder 200 side (e.g., for the same picture).

[0087] The decoded picture may also undergo post-decoding processing (385), such as an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or performing an inverse remapping that is the inverse of the remapping process performed in the pre-encoding process (201). The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream. In an example, the decoded image (e.g., after applying the in-loop filter (365) and / or after the post-decoding processing (385), if post-decoding processing is used) may be transmitted to a display device for presentation to a user.

[0088] Figure 4 4 is a schematic diagram showing an example of a system in which various aspects and examples described herein can be implemented. System 400 may be embodied as a device including various components described below and configured to perform one or more aspects of the aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 400 may be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components, either individually or in combination. For example, in at least one example, the processing and encoder / decoder elements of system 400 are distributed over multiple ICs and / or discrete components. In various examples, system 400 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input ports and / or output ports. In various examples, system 400 is configured to implement one or more aspects of the aspects described in this document.

[0089] The system 400 includes at least one processor 410 configured to execute instructions loaded therein for implementing, for example, various aspects described in this document. The processor 410 may include embedded memory, input-output interfaces, and various other circuits as known in the art. The system 400 includes at least one memory 420 (e.g., a volatile memory device and / or a non-volatile memory device). The system 400 includes a storage device 440, which may include a non-volatile memory and / or a volatile memory, including but not limited to an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a programmable read-only memory (PROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a magnetic disk drive, and / or an optical disk drive. As non-limiting examples, the storage device 440 may include an internal storage device, an attached storage device (including a removable and non-removable storage device), and / or a network-accessible storage device.

[0090] The system 400 includes an encoder / decoder module 430, which is configured to process data, for example, to provide encoded video or decoded video, and the encoder / decoder module 430 may include its own processor and memory. The encoder / decoder module 430 represents a module that may be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both of an encoding module and a decoding module. Additionally, the encoder / decoder module 430 may be implemented as a separate element of the system 400, or may be incorporated into the processor 410 as a combination of hardware and software known to those skilled in the art.

[0091] Program code to be loaded onto the processor 410 or the encoder / decoder 430 to perform various aspects described in this document may be stored in the storage device 440 and subsequently loaded onto the memory 420 for execution by the processor 410. According to various examples, one or more of the processor 410, the memory 420, the storage device 440, and the encoder / decoder module 430 may store one or more of various items during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded video or partially decoded video, bitstreams, matrices, variables, and intermediate or final results of processing equations, formulas, operations, and operation logic.

[0092] In some examples, memory inside the processor 410 and / or the encoder / decoder module 430 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other examples, memory external to the processing device (e.g., the processing device may be the processor 410 or the encoder / decoder module 430) is used for one or more of these functions. The external memory may be a memory 420 and / or a storage device 440, such as a dynamic volatile memory and / or a non-volatile flash memory. In several examples, the external non-volatile flash memory is used to store, for example, an operating system for a television. In at least one example, a fast external dynamic volatile memory (such as RAM) is used as working memory for video encoding and decoding operations.

[0093] Inputs to the elements of system 400 may be provided through various input devices as indicated in block 445. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives an RF signal, such as that transmitted over the air by a broadcaster; (ii) a component (COMP) input terminal (or a set of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Other examples ( Figure 4 ) includes composite video.

[0094] In various examples, the input device of frame 445 has the corresponding input processing element associated as known in the art.For example, the RF part can be associated with the element suitable for the following: (i) select the required frequency (also referred to as selecting signal, or limiting the signal band to a band), (ii) down-convert the selected signal, (iii) again band-limit to a narrower band to select the signal band that (for example) can be called a channel in some examples, (iv) demodulate the signal through down-conversion and band-limited, (v) perform error correction, and / or (vi) demultiplex to select the required data packet stream. The RF part of various examples includes one or more elements for performing these functions, such as frequency selector, signal selector, band limiter, channel selector, filter, down-converter, demodulator, error corrector and demultiplexer. The RF part can include a tuner that performs various functions in these functions, including, for example, down-converting the received signal to a lower frequency (for example, intermediate frequency or near baseband frequency) or to baseband. In a set-top box example, the RF part and its associated input processing element receive the RF signal sent by wired (for example, cable) medium, and filter to the desired frequency band by filtering, down-conversion and again to perform frequency selection.Various examples rearrange the order of above-mentioned (and other) elements, remove some elements in these elements, and / or add other elements that perform similar or different functions.Adding element can include inserting element between existing elements, for example, inserting amplifier and analog-to-digital converter.In various examples, the RF part includes antenna.

[0095] The USB and / or HDMI terminals may include corresponding interface processors for connecting the system 400 to other electronic devices across the USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented as needed, for example, in a separate input processing IC or in the processor 410. Similarly, various aspects of USB or HDMI interface processing may be implemented as needed in a separate interface IC or in the processor 410. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 410 and the encoder / decoder 430, which operate in conjunction with the memory and storage elements to process the data stream as needed for presentation on the output device.

[0096] The various components of system 400 may be disposed within an integrated housing. Within the integrated housing, the various components may be interconnected and transmit data between the components using a suitable connection arrangement 425 (e.g., an internal bus known in the art, including an inter-chip (I2C) bus, wiring, and a printed circuit board).

[0097] System 400 includes a communication interface 450 that enables communication with other devices via a communication channel 460. Communication interface 450 may include, but is not limited to, a transceiver configured to send and receive data through communication channel 460. Communication interface 450 may include, but is not limited to, a modem or a network card, and communication channel 460 may be implemented, for example, within a wired and / or wireless medium.

[0098] In various examples, data is streamed or otherwise provided to the system 400 using a wireless network, such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signals of these examples are received via a communication channel 460 and a communication interface 450 suitable for Wi-Fi communications. The communication channel 460 of these examples is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other cross-top communications. Other examples provide streaming data to the system 400 using a set-top box that delivers data via an HDMI connection of an input box 445. Still other examples use an RF connection of an input box 445 to provide streaming data to the system 400. As described above, various examples provide data in a non-streaming manner. Additionally, various examples use wireless networks other than Wi-Fi, such as cellular networks or network.

[0099] The system 400 may provide output signals to various output devices, including a display 475, a speaker 485, and other peripherals 495. The display 475 of various examples includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 475 may be used for a television, a tablet computer, a laptop, a cellular phone (mobile phone), or other devices. The display 475 may also be integrated with other components (e.g., as in a smart phone), or may be independent (e.g., an external monitor for a laptop computer). In various examples, other peripherals 495 include one or more of an independent digital video disc (or digital versatile disc) (DVD, for both terms), a disc player, a stereo system, and / or a lighting system. Various examples use one or more peripherals 495 that provide functions based on the output of the system 400. For example, a disc player performs the function of playing the output of the system 400.

[0100] In various examples, control signals are transmitted between the system 400 and the display 475, speaker 485, or other peripheral device 495 using signaling such as AV.Link, consumer electronics control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices may be communicatively coupled to the system 400 via dedicated connections through respective interfaces 470, 480, and 490. Alternatively, the output devices may be connected to the system 400 via a communication interface 450 using a communication channel 460. The display 475 and speaker 485 may be integrated into a single unit with other components of the system 400 in an electronic device such as, for example, a television. In various examples, the display interface 470 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0101] For example, if the RF portion of input 445 is part of a separate set-top box, the display 475 and speaker 485 may alternatively be separate from one or more of the other components. In various examples where the display 475 and speaker 485 are external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0102] These examples may be performed by computer software implemented by processor 410 or by hardware or by a combination of hardware and software. As a non-limiting example, these examples may be implemented by one or more integrated circuits. Memory 420 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as, as a non-limiting example, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. Processor 410 may be of any type suitable for the technical environment and may include, as a non-limiting example, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0103] Various specific implementations involve decoding. As used herein, "decoding" may encompass, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various examples, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding.

[0104] As another example, in one example, "decoding" refers only to entropy decoding, in another example, "decoding" refers only to differential decoding, and in another example, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or broadly to a broader decoding process will be clear based on the context of the specific description and is considered to be well understood by those skilled in the art.

[0105] Various specific implementations involve encoding. In a manner similar to the discussion above about "decoding", "encoding" as used in this application can encompass, for example, all or part of the processes performed on an input video sequence to produce an encoded bitstream. In various examples, such processes include one or more of the processes typically performed by an encoder, such as partitioning, differential encoding, transforming, quantization, and entropy encoding.

[0106] As another example, in one example, "encoding" refers only to entropy encoding, in another example, "encoding" refers only to differential encoding, and in another example, "encoding" refers to a combination of differential decoding and entropy encoding. Whether the phrase "encoding process" specifically refers to a subset of operations or broadly refers to a broader encoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0107] It should be noted that syntax elements (e.g., coding syntax, etc.) as used herein are descriptive terms, and these syntax elements may be used, for example, to signal one or more of and / or information associated with one or more of the following: an SDP parameter indicating at least one static 3D region in the V3C content; an RTCP FB message type indicating a 3D region of interest; an SDP parameter indicating an RTCP-based capability to request the desired 3D region; an RTP header extension type indicating the transmitted 3D region information; an SDP parameter indicating an RTP-based capability to signal the transmitted 3D region information; an RTCP FB message type indicating the desired or requested viewport information; and / or an SDP parameter indicating an RTCP-based capability to signal the desired or requested viewport information, etc. Therefore, they do not exclude the use of other syntax element names.

[0108] When the figures are presented as flow charts, it should be understood that they also provide block diagrams of corresponding devices. Similarly, when the figures are presented as block diagrams, it should be understood that they also provide flow charts of corresponding methods / processes.

[0109] The specific implementations and aspects described herein may be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of specific implementation (e.g., discussed only as a method), the specific implementation of the features discussed may also be implemented in other forms (e.g., a device or program). The device may be implemented in, for example, appropriate hardware, software, and firmware. These methods may be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication equipment, such as, for example, a computer, a mobile phone, a portable / personal digital assistant ("PDA"), and other equipment that facilitates information communication between end users.

[0110] References to "one example" or "an example" or "one implementation" or "an implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the example is included in at least one example. Thus, the phrases "in one example" or "in an example" or "in one implementation" or "in an implementation" and any other variations appearing in various places throughout this application are not necessarily all referring to the same example.

[0111] Additionally, the present application may involve "determining" various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory. Obtaining may include receiving, retrieving, constructing, generating, and / or determining.

[0112] Furthermore, the present application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0113] Additionally, the present application may refer to "receiving" various information. Like "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" generally involves, in one way or another, during operations such as, for example, storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0114] It should be understood that, for example, in the cases of "A / B", "A and / or B", and "at least one of A and B", the use of any one of the following, namely " / ", "and / or", and "at least one", is intended to cover the selection of only the first-listed option (A), or only the second-listed option (B), or the selection of both options (A and B). As a further example, in the cases of "A, B, and / or C" and "at least one of A, B, and C", such phrases are intended to cover the selection of only the first-listed option (A), or only the second-listed option (B), or only the third-listed option (C), or the selection of the first-listed option and the second-listed option (A and B), or the selection of the first-listed option and the third-listed option (A and C), or the selection of the second-listed option and the third-listed option (B and C), or the selection of all three options (A and B and C). As will be apparent to those of ordinary skill in the art and related fields, this can be extended to as many items as are listed.

[0115] Moreover, as used herein, the term "signaling" means (among other things) indicating something to a corresponding decoder. The encoder signal can include, for example, one or more of the following: SDP parameters that indicate at least one static 3D region in the V3C content; RTCP FB message types that indicate the 3D region of interest; SDP parameters that indicate the RTCP-based capabilities for requesting the desired 3D region; RTP header extension types that indicate the 3D region information being sent; SDP parameters that indicate the RTP-based capabilities for signaling the 3D region information being sent; RTCP FB message types that indicate the viewport information being desired or requested; and / or SDP parameters that indicate the RTCP-based capabilities for signaling the viewport information being desired or requested, etc. Thus, in one example, the same parameters are used on both the encoder side and the decoder side. Therefore, for example, the encoder can send (explicit signaling) a specific parameter to the decoder such that the decoder can use the same specific parameter. Conversely, if the decoder already has a specific parameter and other parameters, signaling can be used without sending (implicit signaling) to simply allow the decoder to know and select the specific parameter. Bit savings are achieved in various examples by avoiding sending any actual functions. It should be understood that signaling can be implemented in various ways. For example, in various examples, one or more syntax elements, flags, etc. are used to signal information to the corresponding decoder. Although the foregoing relates to the verb form of the term "signaling", the term "signal" can also be used as a noun herein.

[0116] It will be apparent to one of ordinary skill in the art that a specific implementation may generate a variety of signals formatted to carry, for example, storable or transmittable information. The information may include, for example, instructions for executing a method or data generated by one of the described specific implementations. For example, a signal may be formatted to carry a bit stream of the example. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier using the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, a signal may be sent over a variety of different wired or wireless links. The signal may be stored on, or accessed or received from, a processor-readable medium.

[0117] Many examples are described herein. The features of the examples may be provided individually or in any combination across various claim categories and types. In addition, the examples may include one or more of the features, devices or aspects described herein individually or in any combination across various claim categories and types. For example, the features described herein may include a bitstream or signal of information generated as described herein to be implemented. This information may allow a decoder to decode a bitstream, an encoder, a bitstream and / or a decoder according to any of the embodiments described in the embodiments. For example, the features described herein may be implemented by creating and / or sending and / or receiving and / or decoding a bitstream or a signal. For example, the features described herein may be implemented by a method, a process, a device, a medium storing instructions, a medium storing data or a signal. For example, the features described herein may be implemented by a TV, a set-top box, a mobile phone, a tablet computer or other electronic devices that perform decoding. A TV, a set-top box, a mobile phone, a tablet computer or other electronic devices may display (e.g., using a monitor, a screen or other type of display) a result image (e.g., an image reconstructed from a residual of a video bitstream). A TV, a set-top box, a mobile phone, a tablet computer or other electronic devices may receive a signal including a coded image and perform decoding.

[0118] 3D point clouds (e.g., high-quality 3D point clouds) can provide advanced representations of immersive media. A point cloud can include a set of points represented in 3D space by coordinates indicating the location of each point, which can be associated with or accompanied by one or more attributes, such as color, transparency, acquisition time, reflectivity of lasers, or material properties associated with each point. The point cloud can be captured, for example, in one or more (e.g., several) ways. For example, a point cloud can be captured by (e.g., using) multiple cameras and / or depth sensors. A light detection and ranging (LiDAR) laser scanner can (e.g., also) be used to capture a point cloud. The number of points used to reconstruct (e.g., realistically reconstruct) an object and / or scene using a point cloud can be on the order of millions or billions of points. Efficient representation and compression can be used to store and transmit point cloud data.

[0119] Figure 5 An example segmentation of a point cloud is shown.

[0120] Volumetric video (e.g., in uncompressed form) can be represented by a large amount of data. Visual volumetric video based coding (V3C) can exploit the compression efficiency of 2D video codecs to reduce the amount of data used for storage and transmission of volumetric video. The V3C encoder can convert a volume frame into a set of 2D image sequences and associated metadata (referred to as atlas data), which can enable reconstruction of the volume frame. The resulting 2D image sequence can be decoded using a video or image encoder such as Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), or Universal Video Coding (VVC). The atlas data can be decoded, for example, using one or more mechanisms. V3C is a mechanism for capacity video coding. V3C can be used by different applications targeting volumetric content compression, such as point clouds, immersive videos, and / or grid representations of visual volume frames. Examples of such applications may include video-based point cloud compression (V-PCC) and MPEG immersive video (MIV).

[0121] V3C may utilize the High Level Syntax (HLS) syntax (which may also be used for 2D video codecs) to represent the coded atlas data. The coded atlas data may be represented by a Network Abstraction Layer (NAL) unit (eg, a sequence of NALUs).

[0122] A real-time transport protocol (RTP) payload format (eg, for V3C atlas data) may be defined / configured for Network Abstraction Layer (NAL) unit-based 2D video codecs such as H.264 / AVC and HEVC.

[0123] The Session Description Protocol (SDP) may be used to represent information associated with a real-time multimedia and / or communication session. Media capability details, transport addresses, and / or other media description metadata may be exchanged between participants as part of a session negotiation process, for example, when initiating a multimedia conference call, an IP voice call, a video stream, and / or other real-time communication session. The SDP may provide a representation of information regardless of how the information is transmitted. The SDP may be a format for session descriptions. The SDP may not contain a transport protocol.

[0124] An SDP description may include a session-level description section, followed by zero or more media descriptions. The session-level section may begin with a "v=" line, and may continue to the first media description or to the end of the entire description, e.g., whichever comes first. (E.g., each) media description may begin with an "m=" line, and may continue to the next media description or to the end of the entire session description, e.g., whichever comes first. Session-level values ​​may be default values ​​for media present in a session, e.g., unless overridden by an equivalent media-level value.

[0125] SDP may be extended, for example, using attributes. Attributes may be (e.g., defined / configured to be) used as one or more of session-level attributes, media-level attributes, etc. A media description may include zero or more (e.g., any number of) "a=" lines (attribute fields), which may be media description specific and / or may be referred to as media-level attributes that add information about the media description.

[0126] The Real-time Transport Protocol (RTP) can provide end-to-end network transport functionality for applications that send real-time data such as audio, video, and / or analog data over multicast or unicast network services. RTP may not address resource reservation. RTP may not guarantee quality of service for real-time services. Data transmission can be enhanced by the Real-time Control Protocol (RTCP), for example, to enable monitoring of data delivery in a manner that is scalable to large multicast networks, and / or to provide minimal control and identification functions. RTP and / or RTCP can be independent of the underlying transport and network layers.

[0127] The RTP protocol can support the use of RTP-level translators and mixers. The RTP protocol can provide end-to-end delivery services for data with real-time characteristics (such as interactive audio and video). The service may include, for example, one or more of the following: payload type identification, sequence numbering, timestamps, and / or delivery monitoring.

[0128] The RTP protocol may have a header field (e.g., a fixed header field). An RTP packet may include an RTP header (e.g., a fixed RTP header), a contributing source list (which may be empty), and a payload. A control packet (e.g., referred to as an RTCP packet) may include a header (e.g., a fixed header) that may be accompanied by (e.g., followed by) structural elements that vary, for example, depending on the RTCP packet type. The RTP payload may contain data transported by RTP in a packet, such as audio samples and / or compressed video data.

[0129] Table 1 shows an example of the RTP header format. Table 1 - Example of RTP header format

[0130] In some examples, the first twelve octets may be present in each RTP packet. For example, a list of contributing source (CSRC) identifiers may be present if / when (e.g., only if / when) inserted by a mixer. The semantics of the fields in the RTP header may be specified / configured. For example, if the extension bit is set, one or more header extensions may follow the fixed header.

[0131] The RTP header extension mechanism can support (e.g., implemented in an experiment) a function that is independent of the payload format, which can be associated with (e.g., additional) information carried in the RTP data packet header. The RTP header extension mechanism can, for example, be implemented so that the header extension can not be observed (e.g., can be ignored) by other interoperable implementations that have not yet been extended. In one example, for example, if the X bit in the RTP header is set to one, a variable length header extension can be attached to the RTP header (e.g., if present, followed by a CSRC list). Table 2 shows an example of a header extension. Table 2 - Example of header extension

[0132] An RTP header extension can be formed as a sequence of extension elements, for example, with or without padding. An extension element (e.g., each extension element) can have a local identifier and a length. The local identifier can be mapped to a larger namespace in a negotiation (such as session signaling). For example, an RTP header extension can be present in (e.g., can only be present in) a packet if the packet also includes one or more extension elements (e.g., as described herein). For example, an extension element can be present in a packet if / when needed. For example, contrary to indicating that an element actually exists in one or more packets (e.g., all or any packets), signaling of an extension element can indicate that an element (e.g., an element signaled only) can be resent in some packets.

[0133] An extension element (e.g., each extension element) in a packet can have a local identifier (ID) and a length. One or more local identifiers present in a stream may have been negotiated or defined out-of-band. In some examples, there may not be a static assignment of local identifiers. (For example, each) different extension can have a unique ID. A value (e.g., the value zero (0)) can be reserved for padding and / or may not be used as a local identifier. In some examples, there can be multiple (e.g., two) variants of an extension, such as one-byte and two-byte headers.

[0134] A sequence of extension elements (e.g., with or without padding) can form a header extension defined in the RTP specification. There can be as many extension elements as can fit the length indicated in the RTP header extension length. The length can be signaled in whole 32-bit words. Padding bytes can be used to pad boundaries (e.g., 32-bit boundaries). The extension (e.g., the whole extension) can be parsed byte by byte, for example, to find (e.g., each) extension element (e.g., with or without alignment). Parsing can stop earlier at the end of the header extension (e.g., the whole header extension or in a one-byte header), or based on encountering an identifier with a reserved value (e.g., the reserved value is 15). Padding bytes can have the same value (e.g., the value zero (0)), for example, regardless of how parsing ends. Padding bytes can be placed between extension elements (e.g., for alignment) and / or after the last extension element if needed for padding, for example.

[0135] The header extension can be one byte. The value of the header extension (e.g., the 16-bit value of the RTP header extension) can have a fixed bit pattern (e.g., 0xBEDE).

[0136] (For example, each) extension element can start with a byte that includes the ID and the length. Table 3 shows an example of the starting byte of an extension element. Table 3 - Example of starting bytes of an extension element

[0137] As shown in the example in Table 3, a 4-bit ID may be a local identifier for an extension element, which may have a range of 1-14 (inclusive), which may be referred to as a valid range (e.g., for signaling). A local identifier value of 15 may be reserved for future extensions (e.g., rather than being used as an identifier).

[0138] The 4-bit length may be the number of data bytes minus one byte for the header extension element following the one-byte header. A zero value in the field may indicate that one byte of data follows, while a value of 15 (e.g., the maximum value) may indicate 16 bytes of element data. Table 4 shows an exemplary header extension with three extension elements, padding, and other RTP fields. Table 4 - Example of header extension

[0139] The header extension may be two bytes. Table 5 shows an example of a value of the header extension (eg, a 16-bit value of the RTP header extension). Table 5 - Examples of header extension values

[0140] As shown in the example of Table 5, the "appbits" field may be 4 bits, which may be application dependent and / or may be defined to have any value or meaning.

[0141] An extension element (eg, each extension element) may start with a byte including an ID and a byte including a length. Table 6 shows an example of an extension element. Table 6 - Examples of extension elements

[0142] As shown in the example of Table 6, an 8-bit ID may be a local identifier of an element, which may have a range of 1-255 (inclusive). The range 1-256 may be referred to as a valid range, with values ​​1-255 referring to an extension element and value 256 referring to a 4-bit field "appbits" (e.g., as shown above).

[0143] As shown in the example of Table 6, the 8-bit length field may be the length of the extended data in bytes, for example, excluding the ID and length fields. A value of zero (0) may indicate that no data follows.

[0144] Table 7 shows an example of a header extension with three extension elements, padding, and an RTP field. Table 7 - Example of header extension

[0145] A (for example, each) local identifier used in a stream can be mapped to a string using, for example, a property of the following form: a=extmap: <value> [" / " <direction> ] <uri> <extensionattributes> the term <uri>Can be a URI. <value>May be an extended local identifier (ID), which may be an integer included in the valid range. For example, the value zero (0) may be reserved for padding in both forms. For example, the value 15 may be reserved in the single-byte header form as described herein. <direction>Can be indicated by one of "sendonly", "recvonly", "sendrecv", or "inactive".

[0146] A Real-time Transport Control Protocol (RTCP) may be implemented. Real-time media streams using RTP may be resilient (e.g., to some extent) to packet loss. A receiver may use the basic mechanisms of RTCP to report packet reception statistics, which may allow a sender to adjust its transmission behavior in the medium term. RTCP may provide a means (e.g., a unique means) for feedback and / or feedback-based error repair (e.g., in addition to codec-specific mechanisms). An extension may be provided to an Audio-Visual Profile (AVP) that enables a receiver to provide (e.g., statistically) more immediate feedback to a sender, which may allow for short-term adaptation and / or efficient feedback-based repair mechanisms.

[0147] Payload format specific SDP attributes may be defined / configured to indicate the ability to use RTCP feedback (e.g., "a=rtcp-fb"). The "rtcp-fb" attribute may be used (e.g., only) as an SDP media attribute and / or may not be provided at the session level. The "rtcp-fb" attribute may be used (e.g., only) in a media session in which "AVPF" is specified. The "rtcp-fb" attribute may be used to indicate which RTCP FB messages may be used for the indicated payload type in the media session. The wildcard payload type ("*") may be used to indicate that the RTCP feedback attribute applies to all payload types. For example, if several types of feedback are supported and / or if the same feedback is specified for a subset of payload types, several "a=rtcp-fb" lines may be used.

[0148] Payload-specific FB messages can convey information specific to a certain payload type. Payload-specific FB messages can be generated at the codec layer and acted upon. Feedback messages (eg, all feedback messages) may use a common packet format. Table 8 shows an example of a packet format of a feedback message. Table 8 - Example of packet format for feedback message

[0149] As shown in the example in Table 8, fields V, P, SSRC, and length can be defined in the RTP configuration. For example, the version (V) can be two (2) bits. Field V can identify the RTP version (e.g., version 2). Padding (P) can be one (1) bit. Padding (if set) can indicate that the packet includes additional padding octets at the end that are not part of the control information but are included in the length field. The feedback message type (FMT) can be, for example, five (5) bits. The FMT field can identify the type of the FB message. The FMT field can be interpreted relative to the type (e.g., transport layer, payload specific, or application layer feedback). The value of each of the three feedback types can be configured. The payload type (PT) can be, for example, eight (8) bits. The PT field can be an RTCP packet type that identifies the packet as an RTCP FB message. Table 9 shows an example definition of two values. Table 9 - Example definition of PT field values name value Brief Description RTPFB 205 Transport layer FB message PSFB 206 Payload specific FB messages

[0150] The length may be, for example, 16 bits. The length of a packet may be indicated by a 32-bit word minus one, for example including a header and any padding. This length definition may be consistent with the definition of the length field used in RTCP sender and receiver reports. The synchronization source (SSRC) of the packet sender may indicate the SSRC identifier of the originator of the packet. The SSRC of the packet sender may be, for example, 32 bits. The SSRC of the media source may indicate the synchronization source identifier of the media source to which the feedback information segment relates. The SSRC of the media source may be, for example, 32 bits. Feedback control information (FCI) may have a variable length.

[0151] Techniques for capturing and rendering 3D points can enable novel applications in the fields of telepresence, virtual reality, and / or large-scale dynamic 3D maps. 3D Point Cloud Compression (PCC) can include geometry-based compression for static point clouds and video-based compression for dynamic point clouds. 3D PCC can support efficient and interoperable storage and transmission of 3D point clouds. 3D PCC can support lossy and / or lossless decoding of point cloud geometric coordinates and attributes.

[0152] The transmission and signaling of V3C data may be limited to delivery using Dynamic Adaptive Streaming over HTTP (MPEG-DASH) and / or MPEG Media Transport (MMT) protocols.Real-time communication applications may utilize the RTP payload format for visual volumetric video based coding.

[0153] The RTP payload format for visual volumetric video based decoding may specify how to use the RTP protocol to transmit V3C data. The RTP payload format for visual volumetric video based decoding may specify partial access and delivery of V3C data, which may be used for viewport-dependent delivery. For example, based on the user's current viewport to the receiving device, a portion of the point cloud data may be obtained from the sending device. Partial access support may be provided for delivering V3C data using the RTP protocol.

[0154] The RTP / RTCP signaling mechanism can be used to support spatial region-based and / or viewport-based partial access to V3C content. SDP parameters can be used to signal static 3D regions present in immersive media content. RTCP feedback (FB) message types can carry desired 3D regions of interest during RTP transmission of media. RTCP FB messages can be sent from the receiver to the sender with a signal. SDP parameters can indicate RTCP-based capabilities for requesting desired 3D regions, for example, during capability negotiation. RTP header extension types can carry the 3D region information sent during RTP transmission of immersive media. RTP header extension types can be sent from the sender to the receiver with a signal. SDP parameters can indicate RTP-based capabilities for signaling the 3D region information sent, for example, during capability negotiation. SDP parameters can indicate RTP-based capabilities for signaling updated 3D region information, for example, during capability negotiation. RTCP feedback (FB) message types can carry desired and / or requested viewport information, for example, during RTP transmission of media. RTCP FB message types can be sent from the receiver to the sender with a signal. The SDP parameters may indicate, for example, during capability negotiation, an RTCP-based capability to signal desired and / or requested viewport information.

[0155] The streaming may be 3D spatial region dependent streaming.

[0156] Static 3D regions may be signaled in SDP. RTP may be used to send volumetric media. The presence of 3D regions in volumetric media objects may be signaled using Session Description Protocol messages, and client desired 3D regions may be signaled as extensions to Real-time Transport Control Protocol messages (e.g., as defined in RTP / AVPF codec control messages).

[0157] A sender supporting static 3D region mode may provide static 3D region information present in volumetric media in an initial offer-answer negotiation, for example by carrying it in an SDP message. For example, an attribute (e.g., an "a=3d-regions" attribute) may be included below the relevant media line. One or more of the following parameters may be provided in the attributes of a static 3D region (e.g., each static 3D region): region_id; position_x; position_y; position_z; size_X; size_Y; size_Z; or name.

[0158] The parameter region_id may identify a predefined 3D region. The parameter position_x may specify the origin position of the 3D bounding box along the x-axis in Cartesian coordinates. The parameter position_y may specify the origin position of the 3D bounding box along the y-axis in Cartesian coordinates. The parameter position_z may specify the origin position of the 3D bounding box along the z-axis in Cartesian coordinates. The parameter size_X may specify the extension of the 3D bounding box of the volume media along the X-axis in Cartesian coordinates relative to the origin position. The parameter size_Y may specify the extension of the 3D bounding box of the volume media along the Y-axis in Cartesian coordinates relative to the origin position. The parameter size_Z may specify the extension of the 3D bounding box of the volume media along the Z-axis in Cartesian coordinates relative to the origin position. The parameter name may specify the name of the predefined 3D region.

[0159] The syntax of the "a=3d-regions" attribute may conform to the following augmented Backus-Naur form (ABNF):

[0160] For example, you can use the "a=3d-regions" attribute with respect to the media row, as follows: a=3d-regions:99 [region_id=0, position_x=0, position_y=0, position_z=0, size_x=540, size_y=360, size_z=360, name=Head] [region_id=1, position_x=0, position_y=360, position_z=0, size_x=1080, size_y=360, size_z=360, name=Arms] [region_id=2, position_x=0, position_y=720, position_z=0, size_x=540, size_y=360, size_z=360, name=Body] [region_id=3, position_x=0, position_y=1080, position_z=0, size_x=540, size_y=360, size_z=360, name=Legs]

[0161] The RTCP feedback message may include a request for a spatial region, and support for the requested spatial region may be signaled in the SDP message. A client supporting 3D regions may support at least one mode to request a desired 3D region of interest that may be signaled from the receiver to the sender. Modes may include, for example, static 3D regions and / or arbitrary spatial regions.

[0162] Requests for static 3D regions may be provided / received. Clients supporting static 3D region mode may provide static 3D regions in the SDP for (e.g., all) volumetric media streams (e.g., where static 3D region capability is desired). For example, static 3D region capability may be provided by including an a=rtcp-fb attribute with a static 3D region type under the relevant media line scope. A static 3D region type may be represented (e.g., in conjunction with an RTCP feedback method) by the following parameter: 3d-regions-static. A wildcard payload type ("*") may be used to indicate that RTCP feedback attributes for static 3D region mode signaling apply to one or more (e.g., all) payload types. For example, if multiple (e.g., several) types of 3D region signaling are supported and / or if the same static 3D region is specified for a subset of payload types, multiple (e.g., several) "a=rtcp-fb" lines may be used. An example use of the a=rtcp-fb attribute to signal static 3D regions relative to a media line based on an RTCP feedback method may be as follows: a=rtcp-fb:*3d-regions-static

[0163] Requests may be provided / received for arbitrary spatial regions. A client supporting requests for arbitrary spatial regions may indicate such support in an SDP offer for (e.g., all) volumetric media streams (e.g., where arbitrary spatial region request capability is desired). For example, an indication may be provided by including an a=rtcp-fb attribute line within the scope of a relevant media line in an SDP message (e.g., with a feedback message type corresponding to an arbitrary spatial region mode). The RTCP feedback message type corresponding to the arbitrary spatial region mode may be represented by the following parameter: spatial-region-arbitrary. A wildcard payload type ("*") may be used to indicate that the RTCP feedback attribute used to signal an arbitrary spatial region request applies to (e.g., all) payload types. For example, if the same arbitrary spatial region is specified for a subset of payload types, multiple (e.g., several) "a=rtcp-fb" lines may be used. An example of using the a=rtcp-fb attribute to signal support for requests for arbitrary spatial regions in an SDP message based on an RTCP feedback method may be as follows: a=rtcp-fb:*spatial-region-arbitrary

[0164] The SDP offer may include a provided set of static 3D regions, which may be provided using (multiple) RTCP feedback method lines. An RTP session endpoint that accepts the static 3D region request capability in the incoming SDP offer may respond by providing an SDP answer, for example by using (multiple) "a=3d-regions" lines including the accepted set of static 3D regions. The SDP answer may (e.g., also) include an "a=rtcp-fb:*3d-regions-static" line. The accepted set of static 3D regions may be the same as the provided spatial 3D regions or a subset of the provided spatial 3D regions. In some examples, an SDP answer including an "a=rtcp-fb:*3d-regions-static" line but not including an "a=3d-regions" line may indicate that the client supports static 3D region mode and / or that the 3D regions in the provided set of static 3D regions are unacceptable to the client. The recipient may (e.g., after successful negotiation of static 3D regions) request from the accepted set of static 3D regions using an RTCP feedback method. The sender may accordingly encode the sent immersive media to provide the requested static 3D region.

[0165] For example, if the SDP offer provides "a=3d-regions" instead of "a=rtcp-fb:*3d-regions-static", the "a=3d-regions" line may be ignored.

[0166] A new SDP offer-answer negotiation may be performed to modify the static 3D region set. The sender may update the (eg, all) contents of the static 3D regions, including, for example, the total number of static 3D regions and the position, size and / or name of each 3D region.

[0167] RTCP feedback messages may be used to request a 3D region of interest. A 3D region of interest request (or requests) may signal the current 3D region of interest of the immersive media on the receiver side to the sender for (eg, appropriate) encoding and transmission.

[0168] For example, an RTCP feedback message may be used to signal a request for a 3D region of interest, whether the 3D region available on the sender side is static or dynamic. The RTCP feedback message may be identified by a payload type (PT) (e.g., PT equal to 206) that references a payload specific feedback (PSFB) message. The feedback message type (FMT) may be set to a value for a 3D region of interest feedback message (e.g., a value of 18). The RTCP feedback method may involve signaling 3D region of interest information in one or more modes (e.g., immediate feedback and early RTCP modes).

[0169] The RTCP feedback message may be used for static 3D region requests. For example, if / when the 3D region available at the sender is static, the RTCP feedback message for requesting a 3D region of interest may include a region_id parameter. The value of region_id may be obtained from the "a=3d-regions" attribute, which may be signaled during the SDP offer-answer negotiation. For example, two bytes may be used to indicate the value of the region_id parameter. Table 10a shows an example format of the Feedback Control Information (FCI) of the RTCP feedback message for static 3D region requests. Table 10a - Example format of FCI of RTCP feedback message

[0170] For example, if static 3D region and arbitrary spatial region capabilities are successfully negotiated, the RTCP feedback message from the receiver may conform to the message format for static 3D region or arbitrary spatial region, respectively. The sender may distinguish the RTCP feedback message format, for example, by parsing the first 16 bits associated with the mode field. The mode field may be set (e.g., uniquely set) to a value, for example, all values ​​are one in the case of a static 3D region request.

[0171] For example, when the available 3D regions on the sender side are static, the RTCP feedback message for requesting multiple 3D regions of interest may include the number of 3D regions and a list of required region_id parameters. The value of region_id may be obtained from the "a=3d-regions" attribute signaled during the SDP offer-answer negotiation. The value of num_regions and each region_id parameter may be indicated using a set of bytes (e.g., two bytes). Table 10b shows an example format of the FCI of the RTCP feedback message for static 3D region requests.

[0172] Table 10b shows an example where the FCI of the RTCP feedback message for a static 3D region request may be in the following format: Table 10b - Example format of FCI for RTCP feedback message for static 3D region request

[0173] RTCP feedback messages may be used for arbitrary spatial region requests. This document may describe an FCI format for a 3D region of interest. The FCI may include (e.g., exactly one) 3D region. The 3D region information may consist of one or more of the following parameters: position_x, position_y, position_z, size_x, size_y, and / or size_z.

[0174] The parameter position_x may specify the origin position of the 3D bounding box in Cartesian coordinates along the x-axis. The parameter position_y may specify the origin position of the 3D bounding box in Cartesian coordinates along the y-axis. The parameter position_z may specify the origin position of the 3D bounding box in Cartesian coordinates along the z-axis. The parameter size_x may specify the extension of the 3D bounding box of the volumetric media along the x-axis in the Cartesian coordinate system relative to the origin position. The parameter size_y may specify the extension of the 3D bounding box of the volumetric media along the y-axis in the Cartesian coordinate system relative to the origin position. The parameter size_z may specify the extension of the 3D bounding box of the volumetric media along the z-axis in the Cartesian coordinate system relative to the origin position.

[0175] The RTCP feedback message for the 3D region of interest may include, for example, one or more of the parameters position_x, position_y, position_z, size_x, size_y, and / or size_z for an arbitrary spatial region request. For example, four bytes may be used to indicate the value of a parameter (e.g., each parameter). The sender may ignore 3D region of interest requests that describe regions outside of the original volume content. Table 11 shows an example format of an FCI for an RTCP feedback message for an arbitrary spatial region. Table 11 - Example format of FCI for RTCP feedback message for arbitrary spatial region

[0176] Indications of position_x, position_y, position_z, size_x, size_y and / or size_z parameters (e.g., for each four-byte indication) may include a high byte (e.g., indicated by '(h)')), which may be followed by a low byte (e.g., indicated by '(l)') where the low byte holds the least significant bit.

[0177] Streaming that relies on 3D spatial regions may include signaling support for a "Sent 3D Regions" RTP header extension in SDP.

[0178] A client that supports the "static 3D regions" or "arbitrary spatial regions" capability may (e.g., also) include a "sent 3D regions signaling" capability in its SDP offer for (e.g., all) volumetric media streams. A receiver that accepts the "static 3D regions" or "arbitrary spatial regions" capability in an incoming SDP offer may (e.g., also) accept the accompanying "sent 3D regions signaling" offer. A sender that accepts the "static 3D regions" or "arbitrary spatial regions" capability may (e.g., also) accept the accompanying "sent 3D regions" signaling capability offer. For example, the "sent 3D regions" signaling capability may be provided by including an a=extmap attribute, which may indicate the "sent 3D regions" URN under the scope of the associated media line. The "sent 3D regions" signaling URN corresponding to arbitrary spatial regions may be urn:arbitrary-3d-regions-sent. The "sent 3D regions" signaling URN corresponding to static 3D regions may be urn:static-3d-regions-sent. The URN may be used to signal the 3D region being sent relative to a media line (eg, the signaling may be part of an inset component media line), for example, as follows: a=extmap:8urn:arbitrary-3d-regions-sent a=extmap:9urn:static-3d-regions-sent

[0179] For example, if / when a one-byte header extension mechanism is used, the numbers 8 and 9 in the example may be replaced with numbers in the range 1-14. For example, if / when a two-byte header extension mechanism is used, the numbers 8 and 9 in the example may be replaced with numbers in the range 1-254.

[0180] 3D spatial region dependent streaming may include an RTP header extension for signaling the “transmitted 3D region”.

[0181] A response to a 3D region request may involve the sender signaling the sent 3D region, which may be included in the response, which may or may not coincide with the 3D region of interest requested by the recipient, but may include an expanded or reduced version of the requested (one or more) spatial regions, for example, depending on the number and / or size of 3D regions available in the content that overlap with the requested (one or more) spatial regions. The response may help the recipient determine whether and / or when to send subsequent spatial region requests, for example, in response to head movement sensor information and / or based on the spatial volume covered by the 3D region transmitted by the sender. Signaling the 3D region sent by the sender may (e.g., also) indicate the start of an RTP media stream belonging to the requested 3D region of interest.

[0182] For example, if / when the "sent 3D region signaling" capability is successfully negotiated, the sender may signal a response.

[0183] The RTP header extension may be used to signal the transmitted 3D region for a static 3D region request.

[0184] Signaling the sent 3D regions may be done using an RTP header extension, e.g., if the sent 3D region response (e.g., indicated via a URN urn:static-3d-regions-sent during SDP negotiation) corresponds to a request associated with one or more of the static 3D regions, then the RTP header extension may be used to signal the "sent 3D regions". Signaling the sent 3D regions (e.g., using an RTP header extension) may carry num_regions and region_id parameters corresponding to the sent static 3D regions. A two-byte form of the header may be used. Table 12 shows an example two-byte header format using two bytes to indicate the value of num_regions and a list of region_id parameters. Table 12 - Example two-byte header format to indicate the value of num_regions and region_id parameter list In the example shown in Table 12, the Length field can take a value indicating the following number of bytes.

[0185] In some examples, the transmitted 3D region information may be transmitted using a one-byte header. Table 13 shows an exemplary one-byte header format using two bytes to indicate the value of num_regions and a region_id parameter list. Table 13 - Example one-byte header format to indicate the value of num_regions and region_id parameter list In the example shown in Table 13, the Length field can take a value indicating the following number of bytes.

[0186] The RTP header extension may be used to signal the sent 3D region response for arbitrary spatial region requests.

[0187] For example, if the sent 3D region response (e.g., which may be sent via the URN urn:arbitrary-3d-regions-sent in the SDP negotiation) corresponds to an arbitrary spatial region request, then the 3D region sent by the signal may use an RTP header extension. The RTP header extension may carry a num_regions parameter that may specify the number of spatial regions and / or information for each spatial region (spatial region information). (Each) spatial region information may include one or more of the following: for example, parameters position_x, position_y, position_z, size_x, size_y, size_z, region_id, num_tiles, and / or a list of tile identifiers associated with the spatial region. The two-byte form of the header should be used. For example, two bytes may be used to indicate the values ​​of the parameters num_regions, num_tiles, region_id, and / or tile_id. Table 14 shows an example format using four bytes to indicate the values ​​of (e.g., each) parameters position_x, position_y, position_z, size_x, size_y, and / or size_z. Table 14 - Example format using four bytes to indicate parameter values

[0188] As shown in the example in Table 14, the 8-bit ID can be a local identifier. The length field can indicate the number of bytes that follow. Each four-byte indication of the position_x, position_y, position_z, size_x, size_y, and size_z parameters can include a high byte (e.g., indicated by '(h)') followed by a low byte (e.g., indicated by '(l)'). The low byte can hold the least significant bit and the high byte can hold the most significant bit. The num_regiones, num_tiles, region_id, and / or tile_id parameters can be signaled using two bytes (e.g., each), where the high byte (e.g., indicated by '(h)') can be followed by a low byte (e.g., indicated by '(l)').

[0189] The requested 3D region of interest may be indicated as an arbitrary spatial region, for example, via a=rtcp-fb:*spatial-region-arbitrary in the SDP negotiation. The sender may (for example, if the requested 3D region of interest is an arbitrary spatial region) choose to send one or more available spatial regions listed in the SDP corresponding to the requested arbitrary spatial region. The 3D region sent may use the "Static 3D Regions Sent" RTP header extension and / or may carry num_regions and region_id parameters corresponding to static 3D regions sent by the sender in response to a request by the receiver. A two-byte or one-byte form of the header may be used. Table 15 shows an example format with a two-byte header to indicate the values ​​of the num_regions and region_id parameters using two bytes. Table 16 shows an example format with a one-byte header to indicate the values ​​of the num_regions and region_id parameters using two bytes. Table 15 - Example format with a two-byte header to indicate the values ​​of the num_regions and region_id parameters using two bytes

[0190] As shown in the example in Table 15, the Length field can take a value indicating the following number of bytes. Table 16 - Example format with a one-byte header to indicate the values ​​of the num_regions and region_id parameters using two bytes

[0191] For example, arbitrary spatial regions and / or static 3D region request features may be supported bidirectionally and / or unidirectionally, depending on how the client negotiates to support the features during SDP capability negotiation. In some examples, for example, "send only" and / or "receive only" SDP attributes may be used for clients with asymmetric capabilities (e.g., the ability to process 3D region information but not detect / signal 3D region information). Clients may express their one or more capabilities in each direction (e.g., clearly enough) so that signals may be sent in each direction (e.g., only) to the extent that they express useful information and / or can be processed by the receiving party.

[0192] Support may be provided for "arbitrary spatial regions" and / or "static 3D regions" (e.g., only one at a time, or both simultaneously, concurrently, or simultaneously). For example, if / when the sender and receiver successfully negotiate both capabilities, the receiver may decide to request either arbitrary spatial regions or static 3D regions at a given time. For example, if / when the sender provides static 3D regions, the sender may detect and / or track changes to the 3D regions.

[0193] The 3D spatial region dependent streaming may include dynamic 3D region information.

[0194] For example, if / when the 3D region in the immersive media content changes over time, the sender may send (e.g., all) dynamic 3D region information to the receiver when the 3D region is updated or changed. This feature may be referred to as dynamic 3D region support, which is different from the arbitrary spatial region feature. For example, if / when the spatial 3D region configuration in the immersive media content changes over time, a sender that supports dynamic 3D regions may initiate the sent 3D region message. The message may not be sent in response to an RTCP feedback message received from the receiver. The sent 3D region message for an arbitrary spatial region request may be a response from the sender to an RTCP feedback message received from the receiver.

[0195] SDP signaling can be used to send dynamic 3D region information. A sender that supports the "Dynamic 3D regions" capability can (e.g., also) provide the "Sent 3D region signaling" capability in the SDP message for content media content. A receiver that accepts the "Dynamic 3D regions" capability can (e.g., also) accept the "Sent 3D region signaling" capability that accompanies the offer. For example, the "Dynamic 3D regions" capability can be provided by including an a=extmap attribute, which can indicate, for example, the "Sent 3D region signaling" URN under the scope of the relevant media line. The "Sent 3D region signaling" URN corresponding to the dynamic 3D region mode can be urn:dynamic-3d-regions-sent. The URN can be used to signal the sent dynamic 3D regions relative to the media line, for example, as follows: a=extmap:10urn:dynamic-3d-regions-sent

[0196] RTP header extensions may be used for dynamic 3D regions sent. For example, if / when the "3D regions signaling sent" mode corresponds to dynamic 3D regions (which may be indicated via the URN urn:dynamic-3d-regions-sent during SDP negotiation), an RTP header extension may be used to signal updates to the 3D region configuration. The RTP header extension may carry information about the number of spatial regions and / or each spatial region (spatial region information). (Each) spatial region information may include one or more of the parameters position_x, position_y, position_z, size_x, size_y, size_z, region_id, num_tiles, and / or tiles_id lists associated with the spatial region(s). A two-byte form of the header may be used. For example, two bytes may be used to indicate values ​​for parameters num_regions and / or num_tiles. For example, four bytes may be used to indicate values ​​for parameters position_x, position_y, position_z, size_x, size_y, and / or size_z for each spatial region. For example, two bytes may be used to indicate region_id and / or each tile_id.

[0197] Table 17 shows an example format of a transmitted 3D region message in an RTP header extension. Table 18 shows an example format of each 3D region in a transmitted 3D region (eg, as shown in Table 17). Table 17 - Example format for sent 3D region message in RTP header extension Table 18 - Example format for each 3D region in the transmitted 3D region

[0198] For example, if / when the total number of spatial regions is large and / or cannot be accommodated in a single RTP packet, for example due to RTP header extension size limitations and / or RTP packet size limitations, information about (e.g., all) updated spatial regions present in the immersive media content can be signaled over multiple RTP packets.

[0199] Dynamic spatial region information may be sent in multiple RTP packets. For example, if / when the dynamic 3D region feature is supported and the dynamic spatial region information is sent in multiple RTP packets, the "appbits" value may be used to identify the first and last RTP packets that carry the dynamic spatial region information in the RTP header extension data.

[0200] Table 19 shows an example of an RTP header extension with a 16-bit value in a two-byte header form. Table 19 - Example of RTP header extension with 16-bit value in two-byte header form

[0201] Table 20 shows an example of 'appbits' in the RTP header extension for the sent 3D regions message, which may be indicated via the URN urn:dynamic-3d-regions-sent in the SDP negotiation. Table 20 - Example of 'appbits' in RTP header extension for sent 3D region message

[0202] As shown in the example of Table 20, for example, for the first RTP packet carrying dynamic 3D region information, S may be a bit set to one (1). For example, if not, S may be set to zero (0). For example, for the last RTP packet carrying dynamic 3D region information, E may be a bit set to one (1). For example, if not, E may be set to zero (0).

[0203] For example, SDP signaling can be used to provide viewport-based partial access support. A client (e.g., a sender or receiver) that supports streaming immersive media content based on a user's viewport can provide a "viewport-dependent streaming (VDS)" capability in the SDP for (e.g., all) body media content that expects viewport-based immersive media streaming. VDS support can be provided, for example, by including an a=rtcp-fb attribute (e.g., under the relevant media line range). VDS support using the RTCP feedback method can be represented by, for example, the following parameters: 3d-viewport (3D viewport). A wildcard payload type ("*") can be used to indicate that the RTCP feedback attributes used for VDS support signaling apply to (e.g., all) payload types. For example, if / when the same viewport can be specified for a subset of payload types, multiple (e.g., several) "a=rtcp-fb" lines can be used. The a=rtcp-fb attribute can be used to signal viewport-dependent streaming relative to a media line based on the RTCP feedback method, for example as follows: a=rtcp-fb:*3d-viewport

[0204] Clients that support the use of viewport-dependent streaming may include the rtcp-fb:3d-viewport attribute in SDP offers and answers. Clients that support VDS may (e.g., further) support RTCP Feedback (FB) messages to carry requested viewport information during RTP streaming of media (e.g., signaled from the receiver to the sender).

[0205] For example, if the 3d viewport attribute is not available in the SDP offer, the client may not include the 3d viewport attribute in the SDP answer.

[0206] For example, RTCP feedback messages may be used to signal the viewport, thereby providing viewport-based partial access support.

[0207] Viewport-dependent streaming of immersive media content may be successfully negotiated between a sender and a receiver following an SDP-based negotiation process (e.g., as described herein). An RTCP feedback (FB) message 3d-viewport may (e.g., based on successful negotiation) carry the requested viewport information during RTP transmission of the immersive media signaled from the receiver to the sender.

[0208] The signaling of the viewport request may use an RTCP feedback message. The RTCP feedback message may be identified by a payload type (PT) corresponding to a payload specific feedback (PSFB) message (e.g., PT equal to 206). The feedback message type (FMT) may be set to a value "X" for the viewport request feedback message. The RTCP feedback method may involve signaling about viewport information in an immediate feedback and / or early RTCP mode.

[0209] The Feedback Control Information (FCI) format for a 3D viewport request message may be as follows. The FCI may include (e.g., exactly one) 3D viewport. The viewport information in the RTCP feedback viewport message may include, for example, one or more of the following parameters: cam_pos_x; cam_pos_y; cam_pos_z; cam_quat_x; cam_quat_y; cam_quat_z; camera_type; horizontal_fov; vertical_fov; clipping_near_plane; and / or clipping_far_plane.

[0210] The parameters cam_pos_x, cam_pos_y, and cam_pos_z may indicate the x, y, and z coordinates (e.g., in meters) of the position of the camera in the global reference coordinate system, respectively. These values ​​may be expressed in 32-bit binary floating point format, e.g., 4 bytes in big-endian order. The parameters cam_pos_x, cam_pos_y, and cam_pos_z may be associated with the parsing process.

[0211] The parameters cam_quat_x, cam_quat_y, and cam_quat_z may use quaternion representations to indicate the x, y, and z components of the rotation of the camera, respectively. These values ​​may be in the range of -230 to 230 (inclusive). For example, if / when there is no rotation component, the rotation value may be inferred to be zero (0). The value of the rotation component may be calculated, for example, according to Eq (1)-(3): qX=cam_quat_x÷2 30 (1) qY=cam_quat_y÷2 30 (2) qZ=cam_quat_z÷2 30 (3) For example, the fourth component qW of the rotation of the (eg, current) camera model represented using a quaternion can be calculated according to Eq(4): qW=Sqrt(1-(qX2+qY2+qZ2)) (4) A point (w,x,y,z) can represent an angle of rotation about an axis oriented by a vector (x,y,z) such as according to Eq(5): 2*cos^{-1}(w)=2*sin^{-1}(sqrt(x^{2}+y^{2}+z^{2})) (5)

[0212] The parameter camera_type may indicate the projection method of the viewport. ERP projection may be specified, for example, by a value of zero (0). Perspective projection may be specified, for example, by a value of one (1). Orthographic projection may be specified, for example, by a value of two (2). Values ​​in the range of 3 to 255 may be reserved (e.g., for future use).

[0213] The parameter horizontal_fov may indicate (e.g., if / when camera_type is an ERP projection) the longitude range (e.g., in radians) corresponding to the horizontal size of the viewport area. The value may be in the range of 0 to 2π. The parameter horizontal_fov may indicate (e.g., if / when camera_type is a perspective projection) the horizontal field of view (e.g., in radians). The value may be in the range of 0 to π. The parameter horizontal_fov may indicate (e.g., if / when camera_type is an orthographic projection) the horizontal size of the orthographic projection (e.g., in meters). The value may be expressed in a 32-bit binary floating point format, e.g., four (4) bytes in big endian order. The parameter horizontal_fov may be associated with a parsing process.

[0214] The parameter vertical_fov may indicate (e.g., if / when camera_type is ERP projection) the latitude range corresponding to the vertical size of the viewport area, e.g., in radians. The value may be in the range of 0 to π. The parameter vertical_fov may indicate (e.g., if / when camera_type is perspective projection) the relative aspect ratio (e.g., horizontal / vertical) of the viewport for perspective projection. The value may be expressed in 32-bit binary floating point format, e.g., four (4) bytes in big endian order. The parameter vertical_fov may be associated with a parsing process. The parameter vertical_fov may indicate (e.g., if / when camera_type is orthographic projection) the relative aspect ratio of the viewport for orthographic projection (e.g., horizontal / vertical). The value may be expressed in 32-bit binary floating point format, e.g., four (4) bytes in big endian order. The parameter vertical_fov may be associated with a parsing process.

[0215] The parameters clipping_near_plane and clipping_far_plane may indicate near and far depths or distances (e.g., in meters) based on the near and far clipping planes of the viewport, respectively. The values ​​may be expressed in 32-bit binary floating point format, e.g., four (4) bytes in big endian order. The parameters clipping_near_plane and clipping_far_plane may be associated with a parsing process.

[0216] The Interested 3D Viewport Request RTCP feedback message may include, for example, one or more (eg, all) of the following parameters: cam_pos_x, cam_pos_y, cam_pos_z, cam_quat_x, cam_quat_y, and / or cam_quat_z. For example, four bytes may be used to indicate the values ​​of these parameters (eg, each parameter).

[0217] Table 21 shows an example of FCI of the 3D viewport request RTCP feedback message. Table 21 - Example of FCI for 3D Viewport Request RTCP Feedback Message

[0218] In an example, the RTCP viewport feedback message may include one or more flags, such as ext_camera_flag and / or int_camera_flag. These flags may indicate whether external camera parameters (e.g., ext_camera_flag) and / or intrinsic camera parameters (e.g., int_camera_flag) are present in the RTCP message. The message may include a flag, such as equal_fov_flag, which may indicate whether the horizontal FOV and the vertical FOV are equal.

[0219] Table 22 shows an example of FCI of a 3D viewport request RTCP feedback message (eg, including ext_camera_flag, int_camera_flag, and equal_fov_flag). Table 22 - Example of FCI for 3D Viewport Request RTCP Feedback Message

[0220] The RTCP feedback viewport message may include one or more of the following: whether extrinsic camera parameter information is present in the message (e.g., ext_camera_flag); whether the viewport position signaled corresponds to the center of the viewport (e.g., center_view_flag); whether intrinsic camera parameter information is present in the message (e.g., int_camera_flag); whether the horizontal FOV and vertical FOV of the viewport are equal (e.g., equal_FOV_flag); reserved bits (e.g., resv); the projection method of the viewport (e.g., camera_type); the x, y, and / or z coordinates of the camera's position (e.g., c The camera's rotation may be in degrees (e.g., am_pos_x, cam_pos_y, and / or cam_pos_z); the x, y, and / or z components of the camera's rotation expressed using a quaternion (e.g., cam_quat_x, cam_quat_y, and / or cam_quat_z); the longitude extent corresponding to the horizontal size of the viewport area (e.g., horizontal_fov); the latitude extent corresponding to the vertical size of the viewport area (e.g., vertical_fov); or the near and / or far depth or distance based on the viewport's near and far clipping planes (e.g., clipping_near_plane and / or clipping_far_plane).

[0221] The RTCP feedback viewport message may indicate (e.g., include a flag for indicating) that external camera parameter information is present in the message (e.g., ext_camera_flag(E)). The flag may be 1 bit. A flag value equal to 1 may indicate that external camera parameter information is present in the message. A flag value equal to 0 may indicate that external camera parameter information is not present in the message.

[0222] The RTCP feedback viewport message may indicate (e.g., include a flag for indicating) whether the viewport position sent by the signal corresponds to the center of the viewport or to one of the two stereo positions of the viewport (e.g., center_view_flag (C)). The flag may be 1 bit. A flag value equal to 1 may indicate that the viewport position sent by the signal corresponds to the center of the viewport. A flag value equal to 0 may indicate that the viewport position sent by the signal corresponds to one of the two stereo positions of the viewport. In an example, when the external camera parameter flag (e.g., ext_camera_flag) is set to a value of 0, the center viewport position flag (e.g., center_view_flag) value may be set to 0 (e.g., the center viewport position flag may be set to 1 otherwise).

[0223] The RTCP feedback viewport message may indicate (e.g., include a flag indicating) that intrinsic camera parameter information is present in the message (e.g., int_camera_flag(1)). The flag may be 1 bit. A flag value equal to 1 may indicate that intrinsic camera parameter information is present in the message. A flag value equal to 0 may indicate that intrinsic camera parameter information is not present in the message.

[0224] The RTCP feedback viewport message may indicate (e.g., include a flag for indicating) whether the horizontal FOV and vertical FOV of the viewport are equal (e.g., equal_FOV_flag(F)). The flag may be 1 bit. A flag value equal to 1 may indicate that the horizontal FOV and vertical FOV are equal. A flag value equal to 0 may indicate that the horizontal FOV and vertical FOV are not equal. In an example, when the intrinsic camera parameter information is not present in the message (e.g., the int_camera_flag value is 0), the equal FOV flag may be set to 1 (e.g., the equal FOV flag may be set to 1 otherwise).

[0225] The RTCP feedback viewport message may include a reserved bit (eg, resv(R)) for future definition.

[0226] The RTCP feedback viewport message may indicate the projection method of the viewport (e.g., camera_type (CT)). The projection method may be indicated with 3 bits. The indication of the projection method of the viewport may specify one or more of: equirectangular projection (ERP); perspective projection; or orthographic projection. For example, a value of 0 may specify ERP, a value of 1 may specify perspective projection, and / or a value of 2 may specify orthographic projection. In an example, other values ​​(e.g., values ​​in the range of 3 to 7) may be reserved for future use.

[0227] The RTCP feedback viewport message may indicate the x, y, and / or z coordinates of the camera position (e.g., cam_pos_x, cam_pos_y, and / or cam_pos_z, respectively). The x, y, and / or z coordinates of the camera position may be indicated in a global reference coordinate system (e.g., in meters). The values ​​may be expressed in a 32-bit binary floating point format with 4 bytes in big-endian order and with a specified parsing process. The x, y, and / or z coordinates of the camera position may be present in the RTCP feedback viewport message if / when the viewport position signaled corresponds to the center of the viewport (e.g., ext_camera_flag (bit E) is set to 1).

[0228] The RTCP feedback viewport message may indicate the x, y, and / or z components of the rotation of the camera using quaternion representation (e.g., cam_quat_x, cam_quat_y, and / or cam_quat_z, respectively). These values ​​may be in the range of -230 to 230 (inclusive). When the rotation component is not present, its value may be inferred to be equal to 0. In an example, if / when the viewport position sent with the signal corresponds to the center of the viewport (e.g., ext_camera_flag (E bit) is set to 1), the x, y, and / or z components of the rotation of the camera represented using quaternion may be present (e.g., may only be present in this case). The value of the rotation component may be calculated as follows: qX=cam_quat_x÷2 30 ,qY=cam_quat_y÷2 30 ,qZ=cam_quat_z÷2 30

[0229] The fourth component qW of the rotation for the current camera model represented using quaternions can be calculated as follows: qW=Sqrt(1-(qX 2 +qY 2 +qZ 2 ))

[0230] A point (w, x, y, z) can be represented as a rotation of an angle around an axis oriented by a vector (x, y, z): 2*cos^{-1}(w)=2*sin^{-1}(sqrt(x^{2}+y^{2}+z^{2}))

[0231] The RTCP feedback viewport message may indicate the longitude range corresponding to the horizontal size of the viewport area (e.g., horizontal_fov). horizontal_fov may be 32 bits. If / when camera_type is an ERP projection, the value may be expressed in radians and / or may be in the range of 0 to 2π. If / when camera_type is a perspective projection, the value may specify the horizontal field of view (in radians) and / or may be in the range of 0 and π. When camera_type is an orthographic projection, the value may specify the horizontal size of the orthographic projection in meters and / or may be expressed in 32-bit binary floating point format (e.g., in 4-byte big-endian order and with a specified parsing process). This information may be present (e.g., may only be present in this case) when intrinsic camera parameter information is present in the message (e.g., int_camera_flag (bit 1) is set to 1).

[0232] The RTCP feedback viewport message may indicate the latitude range corresponding to the vertical size of the viewport area (e.g., vertical_fov). vertical_fov may be 32 bits. If / when camera_type is ERP projection, the value may be expressed in radians and / or in the range of 0 to π. If / when camera_type is perspective projection, the value may specify the relative aspect ratio (e.g., horizontal / vertical) of the viewport for perspective projection and / or may be expressed in 32-bit binary floating point format (e.g., 4 bytes in big endian and with a specified parsing process). When camera_type is orthographic projection, the value may specify the relative aspect ratio (e.g., horizontal / vertical) of the viewport for orthographic projection and / or may be expressed in 32-bit binary floating point format (e.g., 4 bytes in big endian and with a specified parsing process). This information may be present (e.g., may only be present in this case) when intrinsic camera parameter information is present in the message (e.g., int_camera_flag (bit 1) is set to 1) and when the horizontal FOV and vertical FOV are not equal (e.g., equal_fov_flag (F) is set to 0). In an example, vertical FOV information may not be present.

[0233] The RTCP feedback viewport message may indicate the near and far depths (or distances) (e.g., in meters) based on the near and far clipping planes (e.g., clipping_near_plane and clipping_far_plane) of the viewport. The value may be 32 bits. The value may be expressed in 32-bit binary floating point format (e.g., 4 bytes in big endian order and with a specified parsing process). This information may be present when intrinsic camera parameter information is present in the message (e.g., int_camera_flag (1 bit) is set to 1) (e.g., may only be present in this case).

[0234] Viewport-based partial access support may include an RTP header extension for the sent viewport notification. The sender may receive an RTCP feedback message for the 3D viewport of interest from the receiver. The sender may respond to the receiver (e.g., based on the received message) with one or more 3D spatial regions including the requested 3D viewport. The sent 3D region (one or more) may correspond to a static 3D region, which may be indicated via the URN urn:static-3d-regions-sent in the SDP negotiation. The signaling of the sent 3D region may use an RTP header extension. The signaling may carry num_regions and region_id parameters corresponding to the sent static 3D region (one or more). The format of the RTP header extension of the sent 3D region may be, for example, the format described herein.

[0235] For example, in an ecosystem involving immersive media decoding, real-time streaming of decoded immersive media content, decoding of immersive media on devices, and / or services that can provide low-latency streaming and / or real-time communication of immersive media content, region-of-interest and / or viewport-dependent delivery of V3C data using RTP can be applied.

[0236] Although features and elements are described above in specific combinations, it will be understood by those of ordinary skill in the art that each feature or element may be used alone or in any combination with other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable media include electronic signals (sent via a wired or wireless connection) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media (such as built-in hard disks and removable disks), magneto-optical media, and optical media (such as CD-ROM disks and digital versatile disks (DVDs)). A processor associated with software may be used to implement a radio frequency transceiver for a WTRU, UE, terminal, base station, RNC, or any host computer.< / direction> < / value> < / uri> < / extensionattributes> < / uri> < / direction> < / value>

Claims

1. A device, include: A processor configured to: receiving a Session Description Protocol (SDP) parameter set indicating the presence of one or more 3D regions associated with immersive media content; sending a real-time control protocol (RTCP) feedback (FB) message indicating a three-dimensional (3D) region of interest; as well as One or more 3D regions of visual volumetric video-based coding (V3C) content associated with a 3D region of interest are received.

2. The apparatus of claim 1, wherein the processor is further configured to: Capability negotiation is performed, wherein the capability negotiation is performed using SDP, and wherein the set of SDP parameters is received during the capability negotiation.

3. The apparatus of claim 1 , wherein the processor is further configured to: receiving an SDP message indicating use of a real-time protocol (RTP) header extension, in, The one or more 3D regions of the received V3C content are received using the RTP header extension.

4. The apparatus according to claim 1, wherein the RTCP FB message is sent using an RTCP message. The apparatus of claim 1 , wherein the 3D region of interest is a static 3D region.

6. The device according to claim 5, in, The RTCP FB message includes an indication of a region ID associated with the 3D region of interest.

7. The device according to claim 1, in, The 3D region of interest is an arbitrary 3D region.

8. The device according to claim 7, in, The RTCP FB message includes an indication of a location associated with the 3D region of interest and a size associated with the 3D region of interest.

9. A device comprising a processor, the processor being configured to: receiving a set of SDP parameters indicating the presence of one or more three-dimensional (3D) regions associated with immersive media content; sending a Real-Time Control Protocol (RTCP) Feedback (FB) message indicating the 3D viewport of interest; and One or more 3D regions of visual volumetric video-based coding (V3C) content associated with the 3D viewport of interest are received.

10. The apparatus of claim 9, wherein the processor is further configured to: Capability negotiation is performed, wherein the capability negotiation is performed using SDP, and wherein the set of SDP parameters is received during the capability negotiation. 11 . The apparatus of claim 9 , wherein the one or more 3D regions of the received V3C content are received using a Real Time Protocol (RTP) header extension.

12. The apparatus of claim 9, wherein the RTCP FB message is sent using an RTCP message, and wherein the RTCP FB message further indicates whether there are extrinsic camera parameters associated with the 3D viewport of interest in the RTCP message.

13. The apparatus of claim 9, wherein the RTCP FB message is sent using an RTCP message, and wherein the RTCP FB message further indicates whether intrinsic camera parameters associated with the 3D viewport of interest are present in the RTCP message.

14. The apparatus of claim 13, wherein the RTCP FB message further indicates whether a horizontal FOV associated with the 3D viewport of interest and a vertical FOV associated with the 3D viewport of interest are equal.

15. A method, include: receiving a Session Description Protocol (SDP) parameter set indicating the presence of one or more 3D regions associated with immersive media content; sending a real-time control protocol (RTCP) feedback (FB) message indicating a three-dimensional (3D) region of interest; as well as One or more 3D regions of visual volumetric video-based coding (V3C) content associated with a 3D region of interest are received.

16. The method according to claim 15, further comprising: include: Capability negotiation is performed, wherein the capability negotiation is performed using SDP, and wherein the set of SDP parameters is received during the capability negotiation.

17. The method according to claim 15, in, The SDP parameter set is received using a Real Time Protocol (RTP) header extension.

18. The method of claim 15, wherein the first RTCP FB message is sent using an RTCP message.

19. The method according to claim 15, in, The 3D region of interest is one of a static 3D region or an arbitrary 3D region.

Images