Method and apparatus for picture and / or video coding

Abstract

Methods and apparatus for picture and / or video coding are provided. A method is included, the method comprising: selectively including, in a supplemental enhancement information (SEI) message, sub-picture attribute information for use with adaptive switching of viewports; and including the SEI message in a bitstream.

Description

[0001] This application is a divisional application of the invention patent application with application date of March 11, 2020, application number 202080027777.0, and invention name “Method and device for adaptive resolution change of sub-pictures”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to and the benefit of U.S. Provisional Application No. 62 / 816,686, filed in the U.S. Patent and Trademark Office on March 11, 2019, and U.S. Provisional Application No. 62 / 866,528, filed in the U.S. Patent and Trademark Office on June 25, 2019, the entire contents of each of which are incorporated herein by reference as if fully set forth below for all available purposes. Technical Field

[0004] Embodiments disclosed herein generally relate to methods and apparatus for picture and / or video encoding and decoding in a communication system. Background Art

[0005] This document relates to picture and / or video encoding and decoding for use in communication systems. Summary of the invention

[0006] According to an embodiment of the present disclosure, a method is provided, the method comprising: selectively including sub-picture attribute information for use with adaptive switching of viewports in a Supplemental Enhancement Information (SEI) message; and including the SEI message in a bitstream.

[0007] According to one embodiment of the present disclosure, a method is provided, comprising: receiving a supplemental enhancement information (SEI) message associated with a picture; identifying a set of sub-pictures recommended to perform adaptive resolution change (ARC) based on one or more parameters indicated in the SEI message; and selecting one or more sub-pictures from the set of sub-pictures to perform the ARC.

[0008] According to one embodiment of the present disclosure, a device is provided, comprising one or more processors, wherein the one or more processors are configured to: selectively include sub-picture attribute information for use with adaptive switching of viewports in a supplemental enhancement information (SEI) message; and include the SEI message in a bitstream.

[0009] According to one embodiment of the present disclosure, a device is provided, comprising one or more processors, wherein the one or more processors are configured to: receive a supplemental enhancement information (SEI) message associated with a picture; identify a set of sub-pictures recommended to perform adaptive resolution change (ARC) based on one or more parameters indicated in the SEI message; and select one or more sub-pictures from the set of sub-pictures to perform the ARC. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] A more detailed understanding can be obtained from the following detailed description given by way of example in conjunction with the accompanying drawings. As with the detailed description, the figures in such drawings are examples. Therefore, the drawings and detailed description should not be considered limiting, and other equally effective examples are reasonable and possible. In addition, like reference numerals in the drawings indicate similar elements, and among them:

[0011] Figure 1A is a system diagram illustrating an exemplary communication system in which one or more disclosed embodiments may be implemented;

[0012] Figure 1B It is shown that according to one embodiment, Figure 1A A system diagram of an exemplary wireless transmit / receive unit (WTRU) for use within the communication system shown;

[0013] Figure 1C It is shown that according to one embodiment, Figure 1A A system diagram of an exemplary radio access network (RAN) and an exemplary core network (CN) used within the illustrated communication system;

[0014] Figure 1D It is shown that according to one embodiment, Figure 1A A system diagram of an additional exemplary RAN and an additional exemplary CN used within the illustrated communication system;

[0015] Figure 2 is a block diagram illustrating an example of a video streaming architecture according to one or more embodiments;

[0016] Figure 3 An example of adaptive resolution change (ARC) is shown according to one or more embodiments;

[0017] Figure 4 An example of viewport adaptive streaming according to one or more embodiments is shown;

[0018] Figure 5 An example of viewport switching according to one or more embodiments is shown;

[0019] Figure 6An example of viewport switching with ARC is shown according to one or more embodiments;

[0020] Figure 7 An example of ARC performed based on sub-picture attribute information provided by a Supplemental Enhancement Information (SEI) message according to one or more embodiments is shown;

[0021] Figure 8 illustrates an example of an ARC transition between a low-resolution representation and a high-resolution representation according to one or more embodiments;

[0022] Fig. 9 An exemplary encoding procedure for determining one or more ARC transition points according to one or more embodiments is shown; and

[0023] Fig.10 An example of inter-layer prediction based ARC according to one or more embodiments is shown. DETAILED DESCRIPTION

[0024] In the following detailed description, many specific details are set forth to provide a thorough understanding of the embodiments and / or examples disclosed herein. However, it should be understood that such embodiments and examples can be put into practice without some or all of the specific details set forth herein. In other cases, well-known methods, procedures, components and circuits are not described in detail to avoid blurring the following description. In addition, the embodiments and examples not specifically described herein can be put into practice in place of the embodiments and other examples explicitly, implicitly and / or inherently described, disclosed or otherwise provided (collectively referred to as "providing") herein, or put into practice in combination with these embodiments and examples. Although various embodiments are described and / or claimed herein, wherein devices, systems, equipment, etc. and / or any of their elements perform operations, processes, algorithms, functions, etc. and / or any part thereof, it should be understood that any embodiments described herein and / or protected by the claims assume that any device, system, equipment, etc. and / or any of their elements are configured to perform any operations, processes, algorithms, functions, etc. and / or any part thereof.

[0025] In this application, the terms "reconstruction" and "decoding" can be used interchangeably, the terms "pixel" and "sample" can be used interchangeably, and the terms "image", "picture" and "frame" can be used interchangeably. Usually, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0026] Various methods are described herein, and each method in the method includes one or more steps or actions for implementing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or purposes of specific steps and / or actions may be modified or combined. In addition, terms such as "first", "second", etc. may be used to modify elements, parts, steps, operations, etc., such as "first decoding" and "second decoding" in various embodiments. Unless specifically required, the use of such terms does not imply the ordering of the modified operations. Therefore, in this example, the first decoding does not need to be performed before the second decoding, and may occur, for example, before, during, or during a time period overlapping with the second decoding.

[0027] Representative communication networks

[0028] The methods, devices, and systems provided herein are well suited for communications involving both wired and wireless networks. Wired networks are well known. Figures 1A to 1D An overview of various types of wireless devices and infrastructure is provided, wherein various elements of the network can utilize the methods, apparatuses, and systems provided herein, perform the methods, apparatuses, and systems provided herein, be arranged according to the methods, apparatuses, and systems provided herein, and / or be adapted and / or configured for the methods, apparatuses, and systems provided herein.

[0029] Figure 1A 1 is a schematic diagram illustrating an exemplary communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content such as voice, data, video, messaging, broadcast, etc. to multiple wireless users. The communication system 100 may enable multiple wireless users to access such content through sharing of system resources (including wireless bandwidth). For example, the communication system 100 may employ one or more channel access methods such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single carrier FDMA (SC-FDMA), zero tail unique word DFT spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block filtered OFDM, filter bank multi-carrier (FBMC), etc.

[0030] like Figure 1AAs shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106 / 115, public switched telephone network (PSTN) 108, Internet 110 and other networks 112, but it should be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d (any of which may be referred to as a “station” and / or “STA”) may be configured to transmit and / or receive wireless signals and may include user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular phone, a personal digital assistant (PDA), a smart phone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (IoT) device, a watch or other wearable device, a head-mounted display (HMD), a vehicle, a drone, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in an industrial and / or automated process chain environment), a consumer electronic device, a device operating on a commercial and / or industrial wireless network, etc. Any of the WTRUs 102a, 102b, 102c, and 102d may be interchangeably referred to as a UE.

[0031] The communication system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device that is configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN 106 / 115, the Internet 110, and / or other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node B, an eNode B, a Home Node B, a Home eNode B, a gNB, a New Radio (NR) Node B, a site controller, an access point (AP), a wireless router, and the like. Although the base stations 114a, 114b are each depicted as a single element, it should be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0032] The base station 114a may be part of the RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), a relay node, etc. The base station 114a and / or the base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be in a licensed spectrum, an unlicensed spectrum, or a combination of licensed and unlicensed spectrums. A cell may provide coverage of wireless services to a specific geographic area, which may be relatively fixed or may change over time. The cell may be further divided into cell sectors. For example, a cell associated with the base station 114a may be divided into three sectors. Therefore, in one embodiment, the base station 114a may include three transceivers, for example, one transceiver for each sector of the cell. In one embodiment, the base station 114a may employ multiple-input multiple-output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.

[0033] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0034] More specifically, as noted above, the communication system 100 may be a multiple access system and may employ one or more channel access schemes such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, the base station 114a in the RAN 104 / 113 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may use Wideband CDMA (WCDMA) to establish the air interface 115 / 116 / 117. WCDMA may include communication protocols such as High Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High Speed ​​UL Packet Access (HSUPA).

[0035] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or Advanced LTE (LTE-A) and / or Advanced LTE Pro (LTE-A Pro).

[0036] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as NR radio access, which may establish the air interface 116 using New Radio (NR).

[0037] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may implement LTE radio access and NR radio access together, for example using dual connectivity (DC) principles. Thus, the air interface utilized by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., eNBs and gNBs).

[0038] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement radio technologies such as IEEE 802.11 (e.g., Wireless Fidelity (WiFi)), IEEE 802.16 (e.g., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile Communications (GSM), GSM Enhanced Data Rates for Evolution (EDGE), GSM EDGE (GERAN), etc.

[0039] Figure 1AThe base station 114b in the may be, for example, a wireless router, a Home NodeB, a Home eNodeB, or an access point, and may utilize any suitable RAT to facilitate wireless connectivity in a local area, such as a business location, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a road, and the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish a picocell or a femtocell. As Figure 1A As shown, base station 114b may have a direct connection to Internet 110. Therefore, base station 114b may not need to access Internet 110 via CN 106 / 115.

[0040] The RAN 104 / 113 may be in communication with the CN 106 / 115, which may be any type of network configured to provide voice, data, applications and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have different quality of service (QoS) requirements, such as different throughput requirements, delay requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. The CN 106 / 115 may provide call control, billing services, mobile location-based services, prepaid calling, Internet connectivity, video distribution, etc., and / or perform high-level security functions, such as user authentication. Although not described in detail in the specification, the CN 106 / 115 may be configured to provide voice, data, applications and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. Figure 1A Although not shown in the figure, it will be appreciated that the RAN 104 / 113 and / or the CN 106 / 115 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 104 / 113 or a different RAT. For example, in addition to being connected to the RAN 104 / 113, which may utilize NR radio technology, the CN 106 / 115 may also be in communication with another RAN (not shown) that employs GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0041] The CN 106 / 115 may also act as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network that provides plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the Transmission Control Protocol (TCP), the User Datagram Protocol (UDP), and / or the Internet Protocol (IP) in the TCP / IP Internet protocol suite. The networks 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs, which may employ the same RAT as the RAN 104 / 113 or a different RAT.

[0042] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communication system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). Figure 1A The illustrated WTRU 102c may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[0043] Figure 1B is a system diagram illustrating an exemplary WTRU 102. Figure 1B As shown, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138. It will be appreciated that the WTRU 102 may include any subcombination of the foregoing elements while remaining consistent with an embodiment.

[0044] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal encoding and decoding, data processing, power control, input / output processing, and / or any other functions that enable the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. Although Figure 1B The processor 118 and the transceiver 120 are depicted as separate components, but it is understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0045] The transmit / receive element 122 may be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) via an air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In one embodiment, the transmit / receive element 122 may be a transmitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive RF and light signals. It should be appreciated that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0046] Although the transmit / receive element 122 is Figure 1B 1 as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0047] The transceiver 120 may be configured to modulate signals to be transmitted by the transmit / receive element 122 and to demodulate signals received by the transmit / receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0048] The processor 118 of the WTRU 102 may be coupled to a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit) and may receive user input data therefrom. The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from and store data in any type of suitable memory, such as a non-removable memory 130 and / or a removable memory 132. The non-removable memory 130 may include a random access memory (RAM), a read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from and store data in a memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[0049] The processor 118 may receive power from the power source 134, and may be configured to distribute and / or control power to other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0050] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to or in lieu of the information from the GPS chipset 136, the WTRU 102 may receive location information from a base station (e.g., base stations 114a, 114b) over the air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information by any suitable location-determination method while remaining consistent with an embodiment.

[0051] The processor 118 may also be coupled to other peripherals 138, which may include one or more software modules and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, module, a frequency modulation (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, etc. The peripheral device 138 may include one or more sensors, which may be one or more of the following: a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0052] The WTRU 102 may include a full-duplex radio for which transmission and reception of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous. The full-duplex radio may include an interference management unit 139 for reducing and / or substantially eliminating self-interference via signal processing performed by hardware (e.g., choke) or via a processor (e.g., a separate processor (not shown) or via the processor 118). In one embodiment, the WTRU 102 may include a full-duplex radio for which transmission and reception of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous.

[0053] Figure 1C 1 is a system diagram illustrating the RAN 104 and the CN 106 according to one embodiment. As described above, the RAN 104 may employ an E-UTRA radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 104 may also be in communication with the CN 106.

[0054] The RAN 104 may include evolved Node-Bs 160a, 160b, 160c, though it will be appreciated that the RAN 104 may include any number of evolved Node-Bs while remaining consistent with an embodiment. The evolved Node-Bs 160a, 160b, 160c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the evolved Node-Bs 160a, 160b, 160c may implement MIMO technology. Thus, the evolved Node-B 160a, for example, may use multiple antennas to transmit wireless signals to and / or receive wireless signals from the WTRU 102a.

[0055] Each of the evolved Node Bs 160a, 160b, 160c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the UL and / or DL, etc. Figure 1C As shown, the eNode-Bs 160a, 160b, 160c may communicate with one another via an X2 interface.

[0056] Figure 1C The illustrated CN 106 may include a mobility management entity (MME) 162, a serving gateway (SGW) 164, and a packet data network (PDN) gateway (or PGW) 166. While each of the foregoing elements is depicted as part of the CN 106, it should be understood that any of these elements may be owned and / or operated by an entity other than a CN operator.

[0057] The MME 162 may be connected to each of the evolved Node-Bs 160a, 160b, 160c in the RAN 104 via an S1 interface and may serve as a control node. For example, the MME 162 may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, bearer activation / deactivation, selecting a particular serving gateway during an initial attach of the WTRUs 102a, 102b, 102c, etc. The MME 162 may provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as GSM and / or WCDMA.

[0058] The SGW 164 may be connected to each of the evolved Node-Bs 160a, 160b, 160c in the RAN 104 via an S1 interface. The SGW 164 may generally route and forward user data packets to / from the WTRUs 102a, 102b, 102c. The SGW 164 may perform other functions, such as anchoring the user plane during inter-evolved Node-B handovers, triggering paging when DL data is available for the WTRUs 102a, 102b, 102c, managing and storing the context of the WTRUs 102a, 102b, 102c, and the like.

[0059] The SGW 164 may be connected to the PGW 166, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices.

[0060] The CN 106 may facilitate communications with other networks. For example, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices. For example, the CN 106 may include, or may be in communication with, an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) that serves as an interface between the CN 106 and the PSTN 108. In addition, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers.

[0061] Although the WTRU Figures 1A to 1D Although described as a wireless terminal, it is contemplated that in certain representative embodiments, such a terminal may (eg, temporarily or permanently) use a wired communications interface with a communications network.

[0062] In some representative embodiments, the other network 112 may be a WLAN.

[0063] A WLAN in infrastructure basic service set (BSS) mode may have an access point (AP) for a BSS and one or more stations (STAs) associated with the AP. The AP may have access or an interface to a distribution system (DS) or another type of wired / wireless network that carries traffic to and / or carries traffic away from the BSS. Traffic originating from outside the BSS and leading to the STA may be reached by the AP and may be delivered to the STA. Traffic originating from the STA and leading to a destination outside the BSS may be sent to the AP to be delivered to the corresponding destination. Traffic between STAs within the BSS may be sent by the AP, for example, wherein the source STA may send traffic to the AP, and the AP may deliver the traffic to the destination STA. Traffic between STAs within the BSS may be considered and / or referred to as point-to-point traffic. Point-to-point traffic may be sent between the source and destination STAs (e.g., directly between them) using a direct link setup (DLS). In certain representative embodiments, the DLS may use 802.11e DLS or 802.11z tunnel DLS (TDLS). A WLAN using an independent BSS (IBSS) mode may not have an AP, and STAs (eg, all STAs) within or using the IBSS may communicate directly with each other. The IBSS communication mode may sometimes be referred to herein as an "ad-hoc" communication mode.

[0064] When using the 802.11ac infrastructure operating mode or a similar operating mode, the AP may transmit a beacon on a fixed channel (such as a primary channel). The primary channel may be a fixed width (e.g., a 20 MHz wide bandwidth) or a width dynamically set via signaling. The primary channel may be an operating channel of the BSS and may be used by the STA to establish a connection with the AP. In certain representative embodiments, carrier sense multiple access / collision avoidance (CSMA / CA) may be implemented, for example, in an 802.11 system. For CSMA / CA, a STA (e.g., each STA) (including the AP) may sense the primary channel. If the primary channel is sensed / detected and / or determined to be busy by a particular STA, the particular STA may back off. One STA (e.g., only one station) may transmit at any given time in a given BSS.

[0065] High throughput (HT) STAs may communicate using a 40 MHz wide channel, for example, via a combination of a primary 20 MHz channel and adjacent or non-adjacent 20 MHz channels to form a 40 MHz wide channel.

[0066] Very high throughput (VHT) STA can support 20MHz, 40MHz, 80MHz and / or 160MHz wide channels. 40MHz and / or 80MHz channels can be formed by combining continuous 20MHz channels. A 160MHz channel can be formed by combining 8 continuous 20MHz channels, or by combining two non-continuous 80MHz channels (this can be called 80+80 configuration). For the 80+80 configuration, after channel coding, the data can pass through a segment parser that can divide the data into two streams. Each stream can be processed by inverse fast Fourier transform (IFFT) and time domain processing separately. These streams can be mapped to two 80MHz channels, and data can be transmitted by transmitting STA. At the receiver of the receiving STA, the above-mentioned operation for the 80+80 configuration can be reversed, and the combined data can be sent to the medium access control (MAC).

[0067] 802.11af and 802.11ah support operating modes below 1GHz. Channel operating bandwidths and carriers are reduced in 802.11af and 802.11ah relative to those used in 802.11n and 802.11ac. 802.11af supports 5MHz, 10MHz, and 20MHz bandwidths in the TV White Space (TVWS) spectrum, and 802.11ah supports 1MHz, 2MHz, 4MHz, 8MHz, and 16MHz bandwidths using non-TVWS spectrum. According to a representative embodiment, 802.11ah may support instrument type control / machine type communications, such as MTC devices in macro coverage areas. MTC devices may have certain capabilities, such as limited capabilities, including support for (e.g., only support for) certain bandwidths and / or limited bandwidths. MTC devices may include batteries with battery life above a threshold (e.g., to maintain very long battery life).

[0068] WLAN systems that can support multiple channels and channel bandwidths such as 802.11n, 802.11ac, 802.11af, and 802.11ah include channels that can be designated as primary channels. The primary channel may have a bandwidth equal to the maximum common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel may be set and / or limited by a STA (which supports the minimum bandwidth operating mode) from all STAs operating in the BSS. In the example of 802.11ah, for STAs (e.g., MTC-type devices) that support (e.g., only support) 1MHz mode, the primary channel may be 1MHz wide, even if the AP and other STAs in the BSS support 2MHz, 4MHz, 8MHz, 16MHz, and / or other channel bandwidth operating modes. Carrier sensing and / or network allocation vector (NAV) settings may depend on the state of the primary channel. If the primary channel is busy, for example, because a STA (supporting only the 1MHz operating mode) is transmitting to the AP, the entire available frequency band may be considered busy even if most of the frequency bands remain idle and may be available.

[0069] In the United States, the available frequency band for 802.11ah is 902MHz to 928MHz. In South Korea, the available frequency band is 917.5MHz to 923.5MHz. In Japan, the available frequency band is 916.5MHz to 927.5MHz. The total bandwidth available for 802.11ah is 6MHz to 26MHz, depending on the country code.

[0070] Figure 1D 1 is a system diagram showing the RAN 113 and the CN 115 according to one embodiment. As noted above, the RAN 113 may employ NR radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 113 may also be in communication with the CN 115.

[0071] The RAN 113 may include gNBs 180a, 180b, 180c, though it will be appreciated that the RAN 113 may include any number of gNBs while remaining consistent with an embodiment. The gNBs 180a, 180b, 180c may each include one or more transceivers to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the gNBs 180a, 180b, 180c may implement MIMO technology. For example, the gNBs 180a, 180b may utilize beamforming to transmit signals to and / or receive signals from the gNBs 180a, 180b, 180c. Thus, the gNB 180a may, for example, use multiple antennas to transmit wireless signals to and / or receive wireless signals from the WTRU 102a. In one embodiment, the gNBs 180a, 180b, 180c may implement carrier aggregation techniques. For example, the gNB 180a may transmit multiple component carriers to the WTRU 102a (not shown). A subset of these component carriers may be on unlicensed spectrum, while the remaining component carriers may be on licensed spectrum. In one embodiment, the gNBs 180a, 180b, 180c may implement coordinated multi-point (CoMP) techniques. For example, the WTRU 102a may receive coordinated transmissions from gNB 180a and gNB 180b (and / or gNB 180c).

[0072] The WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using transmissions associated with scalable parameter sets. For example, OFDM symbol spacing and / or OFDM subcarrier spacing may vary for different transmissions, different cells, and / or different portions of the wireless transmission spectrum. The WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using subframes or transmission time intervals (TTIs) of varying or scalable lengths (e.g., containing different numbers of OFDM symbols and / or continuously varying absolute time lengths).

[0073] The gNBs 180a, 180b, 180c may be configured to communicate with the WTRUs 102a, 102b, 102c in a standalone configuration and / or a non-standalone configuration. In a standalone configuration, the WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c while also not accessing other RANs (e.g., such as the evolved Node-Bs 160a, 160b, 160c). In a standalone configuration, the WTRUs 102a, 102b, 102c may use one or more of the gNBs 180a, 180b, 180c as mobility anchor points. In a standalone configuration, the WTRUs 102a, 102b, 102c may communicate with the gNBs 180a, 180b, 180c using signals in an unlicensed band. In a non-standalone configuration, the WTRUs 102a, 102b, 102c may communicate or connect with the gNBs 180a, 180b, 180c while also communicating or connecting with other RANs, such as the eNode-Bs 160a, 160b, 160c. For example, the WTRUs 102a, 102b, 102c may implement the DC principle to communicate with one or more gNBs 180a, 180b, 180c and one or more evolved Node-Bs 160a, 160b, 160c substantially simultaneously. In a non-standalone configuration, the evolved Node-Bs 160a, 160b, 160c may serve as mobility anchors for the WTRUs 102a, 102b, 102c, and the gNBs 180a, 180b, 180c may provide additional coverage and / or throughput for serving the WTRUs 102a, 102b, 102c.

[0074] Each of the gNBs 180a, 180b, 180c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in UL and / or DL, support of network slicing, dual connectivity, interworking between NR and E-UTRA, routing of user plane data towards a user plane function (UPF) 184a, 184b, routing of control plane information towards an access and mobility management function (AMF) 182a, 182b, etc. Figure 1D As shown, gNBs 180a, 180b, and 180c may communicate with each other via an Xn interface.

[0075] Figure 1DThe CN 115 shown may include at least one AMF 182a, 182b, at least one UPF 184a, 184b, at least one session management function (SMF) 183a, 183b, and possible data networks (DNs) 185a, 185b. Although each of the foregoing elements is depicted as part of the CN 115, it should be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0076] The AMF 182a, 182b may be connected to one or more of the gNBs 180a, 180b, 180c via the N2 interface in the RAN 113 and may serve as a control node. For example, the AMF 182a, 182b may be responsible for authenticating users of the WTRU 102a, 102b, 102c, support of network slicing (e.g., handling of different PDU sessions with different requirements), selecting a specific SMF 183a, 183b, management of registration areas, termination of NAS signaling, mobility management, etc. The AMF 182a, 182b may use network slicing in order to customize CN support for the WTRU 102a, 102b, 102c based on the type of services used by the WTRU 102a, 102b, 102c. For example, different network slices may be established for different use cases, such as services relying on ultra-reliable low latency (URLLC) access, services relying on enhanced mobile broadband (eMBB) access, services for machine type communication (MTC) access, etc. The AMF 182 may provide a control plane function for switching between the RAN 113 and other RANs (not shown) that employ other radio technologies, such as LTE, LTE-A, LTE-APro, and / or non-3GPP access technologies, such as WiFi.

[0077] The SMF 183a, 183b may be connected to the AMF 182a, 182b in the CN 115 via the N11 interface. The SMF 183a, 183b may also be connected to the UPF 184a, 184b in the CN 115 via the N4 interface. The SMF 183a, 183b may select and control the UPF 184a, 184b, and configure traffic routing through the UPF 184a, 184b. The SMF 183a, 183b may perform other functions such as managing and allocating WTRU or UE IP addresses, managing PDU sessions, controlling policy enforcement and QoS, providing downlink data notification, etc. The PDU session type may be IP-based, non-IP-based, Ethernet-based, etc.

[0078] The UPF 184a, 184b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via the N3 interface, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices. The UPF 184, 184b may perform other functions, such as routing and forwarding packets, enforcing user plane policies, supporting multi-homed PDU sessions, handling user plane QoS, buffering downlink packets, providing mobility anchoring, etc.

[0079] The CN 115 may facilitate communications with other networks. For example, the CN 115 may include or may communicate with an IP gateway (e.g., an IP Multimedia Subsystem (IMS) server) that serves as an interface between the CN 115 and the PSTN 108. In addition, the CN 115 may provide the WTRUs 102a, 102b, 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers. In one embodiment, the WTRUs 102a, 102b, 102c may be connected to local data networks (DNs) 185a, 185b via the UPF 184a, 184b via an N3 interface to the UPF 184a, 184b and an N6 interface between the UPF 184a, 184b and the DN 185a, 185b.

[0080] Given that Figures 1A to 1D as well as Figures 1A to 1D Corresponding to the description of the present invention, one or more or all of the functions described herein with reference to one or more of the following may be performed by one or more simulation devices (not shown): WTRU102a-d, base station 114a-b, evolved Node B 160a-c, MME 162, SGW 164, PGW 166, gNB 180a-c, AMF182a-b, UPF 184a-b, SMF 183a-b, DN 185a-b and / or any other device described herein. The simulation device may be one or more devices configured to emulate one or more or all of the functions described herein. For example, the simulation device may be used to test other devices and / or simulate network and / or WTRU functions.

[0081] The simulation device may be designed to implement one or more tests of other devices in a laboratory environment and / or an operator network environment. For example, the one or more simulation devices may perform one or more or all functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network in order to test other devices within the communication network. The one or more simulation devices may perform one or more functions or all functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. The simulation device may be directly coupled to another device for testing purposes and / or may use over-the-air wireless communications to perform testing.

[0082] The one or more simulation devices may perform one or more (including all) functions without being implemented / deployed as part of a wired and / or wireless communication network. For example, the simulation device may be used in a test scenario in a test lab and / or a non-deployed (e.g., testing) wired and / or wireless communication network to implement testing of one or more components. The one or more simulation devices may be test equipment. Direct RF coupling and / or wireless communication via RF circuits (e.g., which may include one or more antennas) may be used by the simulation device to transmit and / or receive data.

[0083] Representative architecture / framework

[0084] Video codec systems can be used to compress digital video signals, which can reduce storage requirements and / or transmission bandwidth of video signals. Video codec systems may include block-based, wavelet-based and / or object-based systems. Block-based video codec systems can be based on one or more standards, use one or more standards, follow one or more standards, comply with one or more standards, etc., and the one or more standards are such as MPEG-1 / 2 / 4 Part 2, H.264 / MPEG-4 Part 10 AVC, VC-1, High Efficiency Video Codec (HEVC) and / or Versatile Video Codec (VVC). Block-based video codec systems may include a block-based hybrid video codec framework.

[0085] Figure 22 is a block diagram illustrating an example of a video streaming architecture 200. In this example, a server 202 may include one or more video encoders (e.g., encoders 204, 206, and 208), each of which may generate a video bitstream at a different resolution, frame rate, or bitrate. An intermediate block 210 may be used or configured. In one example, the intermediate block 210 may be a media aware network element (MANE). The intermediate block 210 may generate, forward, identify, or parse a high-level syntax of an input video bitstream, extract a sub-bitstream from one input video bitstream, and / or output the extracted sub-bitstream to a client or decoder 212. The intermediate block 210 may extract multiple sub-bitstreams from multiple input video bitstreams and combine them together to form a new output video bitstream delivered to the client or decoder 212.

[0086] In various embodiments, one or more encoders (e.g., encoders 204, 206, and / or 208), intermediate block 210, and / or decoder 212 may be implemented in a device having a processor communicatively coupled to a memory. The memory may include instructions executable by the processor, including instructions for performing any of the various embodiments (e.g., representative procedures) disclosed herein. In various embodiments, the device may be configured as and / or configured with various elements of a wireless transmit and receive unit (WTRU). Figures 1A to 1D Exemplary details of the WTRU and its elements are provided in the and accompanying disclosures.

[0087] Various methods and aspects described in this application can be used to modify modules, such as Figure 2 Intra-prediction, entropy coding and / or decoding modules of one or more video encoders (e.g., encoders 204, 206, and 208) and decoders 212 are shown. In addition, aspects of the present invention are not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations (whether pre-existing or developed in the future) and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this application can be used alone or in combination.

[0088] Various numerical values ​​are used in this application. The specific values ​​are for exemplary purposes, and the described aspects are not limited to these specific values.

[0089] Representative procedures for adaptive procedures with varying resolution

[0090] Various schemes using AVC and / or HEVC may not have the ability to change resolution without introducing intra random access point (IRAP) pictures. IRAP pictures that are encoded and decoded with reasonable quality typically have a much larger frame size (e.g., a greater number of bits used to encode and decode the frame) than non-IRAP pictures. In addition, IRAP pictures are more complex to decode. Adaptive resolution change (ARC) may refer to any of the schemes and capabilities in which the spatial resolution may be changed at non-IRAP pictures.

[0091] Figure 3 An example of an ARC mechanism 300 is shown. Figure 3 , high-resolution picture frame No. 3 may be encoded using inter-frame prediction from reference picture frame No. 2 having the same resolution, and low-resolution picture frame No. 5 may be encoded using inter-frame prediction from reference picture frame No. 4 having the same resolution. During ARC, high-resolution picture frame No. 3 may be reconstructed from reference picture frame No. 2, which is upscaled from low-resolution picture frame No. 2 for motion compensation, and low-resolution picture frame No. 5 may be reconstructed from reference picture frame No. 4, which is downscaled from high-resolution picture frame No. 4 for motion compensation. Thus, resolution switching may occur or be performed on one or more non-IRAP frames.

[0092] In various specific implementations, for example, a multi-party video conference may benefit from using ARC to process picture and / or video ("picture / video") frames, where one or more or all participants (i.e., their pictures / videos) are displayed individually on a shared screen, and the active speaker (i.e., his picture / video) is displayed at a larger video size than the remaining participants. If the active speaker changes frequently, ARC may be used (e.g., it is desirable to use ARC) to efficiently implement frequent and / or unpredictable resolution changes due to swapping in new active speakers and swapping out old active speakers. Current adaptive video streaming methods typically change the video representation bitrate or resolution after an IRAP picture to match different network bandwidths.

[0093] ARC can improve adaptive streaming performance by eliminating the need to send one or more high (or large) frame size IRAP pictures. ARC can reduce streaming start delays because applications typically buffer up to a certain number of decoded pictures and / or a certain range of decoding times before display, and for example, taking into account smaller sized pictures. Under current viewport adaptive 360-degree video streaming based on motion constraint tile sets (MCTS), sub-pictures representing the viewport are typically delivered using high resolution, and sub-pictures representing other areas (e.g., areas not in the user's field of view) are typically delivered using lower resolutions. When the viewport changes, the corresponding resolution of the sub-picture changes accordingly, and the user experience is affected by the switching delay of the high-quality viewport.

[0094] Figure 4 An example of a viewport adaptive streaming mechanism 400 is shown. A viewport sub-picture (e.g., a front view) is extracted from a large resolution 360-degree video (represented by 1), and sub-pictures representing other areas are extracted from a low resolution 360-degree video (represented by 2). The extracted sub-pictures may be combined into a single representation (e.g., as Figure 4 The resulting composed or merged viewport adaptive video is delivered to the user (or client) so that the user can experience a high-quality viewport with a reduced delivery bandwidth. When the user changes the viewport from the front view to the right view, a high-resolution right view sub-picture is extracted on the IRAP picture, and a new video frame of the composed or merged video combining the high-resolution right view sub-picture and the low-resolution front sub-picture is formed. Therefore, the length of the IRAP distance can affect the high-quality viewport switching delay, and high-bitrate IRAP pictures can also increase the network and / or processing load. ARC can achieve faster viewport switching and can support different IRAP distances for different (e.g., 360-degree) video representations.

[0095] A picture / video frame may be predicted across resolutions by rescaling a reference picture (e.g., using the methods proposed in [1] and / or [2]). A picture resolution index (PRI) may be signaled in a picture parameter set (PPS) to indicate that a slice (e.g., associated with a picture) is from a picture and has a resolution indicated by the index. It may be desirable to allow a sub-picture derived from a random access (e.g., IRAP) picture and another sub-picture derived from a non-random access (e.g., non-IRAP) picture to be merged into the same codec picture conforming to the generic video codec (VVC) (e.g., using the scheme proposed in [3]).

[0096] Representative procedures for viewport switching

[0097] For viewport adaptive streaming (e.g., in conjunction with 360-degree video), a sub-bitstream corresponding to each sub-picture can be extracted from its original bitstream and / or representation, and multiple sub-bitstreams can be merged to form a new bitstream. The original bitstream and / or the new bitstream can be, for example, a bitstream of HEVC, VVC, and / or similar type. In current viewport adaptive streaming, sub-bitstream merging is performed in the compressed domain, and doing so may introduce several problems. For example, viewport switching only occurs when all involved sub-pictures are instantaneous decoding refresh (IDR) pictures to ensure that the non-IRAP pictures that follow (in time) have the correct reference pictures and / or reference sub-pictures. However, using IDR pictures may introduce latency issues.

[0098] Figure 5A viewport switching mechanism 500 is shown, in which a current viewport is switched to a new viewport. Figure 5 , the right view sub-picture undergoes switching from lower resolution to higher resolution, and the front view sub-picture undergoes switching from higher resolution to lower resolution to match the new viewport. The dotted line represents temporal inter-frame prediction. In the current viewport switching scheme, switching can only occur when the right view sub-picture at higher resolution and the front view sub-picture at lower resolution are both IDR pictures, because subsequent sub-pictures of the same view are inter-predicted from the same resolution sub-picture. Other sub-pictures, such as top view, back view, left view, and bottom view, do not have to be encoded as IRAP pictures because inter-frame prediction continues on sub-pictures with the same resolution.

[0099] According to the methods and / or techniques provided herein, ARC may be performed in conjunction with adaptive viewpoint switching (and / or adaptive viewport streaming), such that, for example, sub-pictures may be predicted from sub-pictures of the same view at different resolutions. Figure 6 An example of a viewport switching mechanism 600 with ARC is shown. Figure 6 , a high-resolution right view sub-picture can be predicted from a previous low-resolution right view sub-picture, and a low-resolution front view sub-picture can be predicted from a previous high-resolution front view sub-picture. By using the previous sub-picture, the sub-picture to be predicted can be predicted without introducing an IRAP sub-picture. The switching delay and the transmission bit rate can be reduced by not introducing an IRAP sub-picture.

[0100] Currently, VVC does not specify a decoding process for such sub-picture extraction and repositioning schemes, including when sub-pictures of the same view may be packed at different locations within the picture (e.g., locations that change at certain times). The motion vector of each sub-picture may provide an offset from the coordinates in the decoded sub-picture to the coordinates in the reference sub-picture, and the coordinates should be (or at least can be assumed to be) consistent across the entire picture with the same resolution.

[0101] Still see Figure 6 , the reference sub-picture can be scaled up or down to match the resolution of the current sub-picture, but the coordinates of the reference sub-picture and the current sub-picture within the picture may be different. According to the methods and / or techniques provided herein, the current decoding process and associated signaling can be modified to implement and / or implement the viewport adaptation method disclosed herein. The methods and / or techniques provided herein solve defects in current signaling associated with decoding, including the following defects: For example, at the basic bitstream level or the system level, there is no signaling or metadata to identify a set or group of sub-pictures of multiple representations mapped to the same two-dimensional (2D) or three-dimensional (3D) content area.

[0102] Several supplemental enhancement information (SEI) messages specify rectangular regions and 360-degree video information in HEVC. The pan scan rectangle SEI message specifies the coordinates of one or more rectangular regions relative to the consistent cropping window specified by the active SPS. The equirectangular and cube map projection SEI messages provide information to enable the color samples of the projected picture to be remapped to a spherical coordinate space (e.g., using spherical coordinate addressing) to support 360-degree video pictures. The region-wise packing SEI message provides information to enable the color samples of the cropped decoded picture to be remapped to the projected picture, as well as information about the location and size of the guard band (if any). However, all of these SEI messages are designed for a single representation (or layer) and therefore do not address relationships between sub-pictures across multiple representations with different resolutions.

[0103] Representative procedures for sub-picture-based applications

[0104] For some sub-picture-based applications, each extracted sub-picture can be assigned to a different position in the new picture. As used herein, a sub-picture attribute SEI message refers to an SEI message including sub-picture attribute information, which may indicate one or more sub-pictures or tile groups across multiple layers or representations associated with the same source content area (and / or define attributes that define or indicate the one or more sub-pictures or tile groups). In one embodiment, the sub-picture attribute information may include one or more additional indicators to indicate one or more recommended ARC switching points. Alternatively, a sub-picture attribute SEI message or a similar type of SEI message may include an indicator to indicate one or more recommended ARC switching points. In one embodiment, the sub-picture attribute information may include one or more actual ARC switching points, for example, to achieve better reconstructed picture quality or to apply certain constraints. Alternatively, a sub-picture attribute SEI message or a similar type of SEI message may include one or more actual ARC switching points.

[0105] Sub-picture attributes SEI message

[0106] In one embodiment, the video content can be encoded into multiple codec versions or representations (or layers). Each representation can be encoded and decoded with different resolutions and / or qualities. For example, for 360-degree video, each representation can be a different projection and / or regional packaging format. The original content area can be mapped to different parts of the representation, i.e., sub-pictures. The sub-pictures can be rotated, scaled or projected differently in different representations. The position and size of the sub-pictures corresponding to the same content area can also be changed across different representations. The middle frame or client can obtain one or more sub-pictures across representations. The middle frame or client can use multiple obtained sub-pictures to form new pictures for viewport-related streaming applications. The new picture can combine different sub-pictures with different quality levels and / or resolutions. For example, new picture formation can be performed to meet the streaming needs of (e.g., 360-degree) video clients.

[0107] In one embodiment, a content producer may generate multiple representations. All representations associated with the same content may be configured into a multi-layer structure. In one embodiment, each representation may be a layer. Each layer may be independently coded or may depend on other layers. A layer ID may be used to identify a specific coded video representation. Each layer may have multiple sub-pictures. Each sub-picture may be identified by a unique sub-picture ID or a tile group ID. The correspondence between the sub-picture and the original content area (e.g., an area on a 360-degree content sphere) may be derived from the projection and area-wise SEI messages associated with each representation. However, doing so will require the middle box or client to parse multiple SEI messages from each layer, and the export process may increase the workload of the middle box or client. A single SEI message or parameter set describing sub-picture attributes (e.g., the correspondence between sub-pictures across multiple layers, and the mapping of sub-pictures in each layer to areas on a (e.g., 360-degree) video sphere) may simplify the mapping between sub-pictures and corresponding original content areas for applications based on sub-pictures.

[0108] In one embodiment, the sub-picture partitioning layout can be signaled in the PPS. The sub-picture resolution can be explicitly signaled (e.g., in the PPS or tile group header). Alternatively, the sub-picture resolution can be derived from the tile group layout and the entire picture resolution. In order to map (e.g., each) sub-picture to the corresponding spherical space, the sub-picture attribute SEI message may include and / or provide information such as the layer ID, tile group ID, the coordinates of the sub-picture and its mapping on the spherical coordinate space. The SEI message may list any or all sub-pictures that can be used for viewport adaptive streaming and regional packaging of sub-pictures. The decoder can identify corresponding reference sub-pictures that may or may not be juxtaposed with the current sub-picture based on such SEI messages. Depending on the resolution of the reference sub-picture, the decoder can scale the sub-picture for ARC and align the coordinates between the current sub-picture and the reference sub-picture.

[0109] Table 1 provides an example of the syntax structure of the sub-picture attribute SEI message. Table 1 lists the number of spherical regions (viewports) and the sub-pictures covering the same spherical region. Table 1 also provides the coordinates of each repositioned sub-picture relative to the consistent cropping window specified by the active sequence parameter set (SPS).

[0110] Table 1 – Example of a sub-picture attributes SEI message

[0111]

[0112] In Table 1, num_source_content_regions_minus1 plus 1 may specify the number of source content regions specified by the SEI message.

[0113] In Table 1, source_content_region_position may specify the position of the i-th source content region. For 2D source content, it may be the top left position of the region in 2D coordinates. For 360-degree video content, it may be the sphere center azimuth and tilt position.

[0114] In Table 1, source_content_region_size may specify the size of the i-th source content region. For 2D source content, it may be the width and height of the region. For 360-degree video content, it may be the tilt angle in degrees of a spherical region passing through the center point of the spherical region relative to the global coordinate axes, the azimuth range, and the elevation range.

[0115] In Table 1, num_subpics_minus1[i] plus 1 may specify the number of sub-pictures associated with the i-th source content region.

[0116] In Table 1, layer_id[i][j] may specify the layer identifier to which the j-th sub-picture associated with the i-th source content region belongs.

[0117] In Table 1, subpic_coordinate[i][j] may specify the coordinates of the jth sub-picture associated with the i-th source content region within the picture. It may be the upper left position of the sub-picture or the center position of the sub-picture.

[0118] In Table 1, subpic_id[i][j] may specify the subpicture ID of the j-th subpicture associated with the i-th source content region. The subpicture ID and / or the tile group ID may be used to identify the subpicture.

[0119] In Table 1, subpic_width[i][j] and subpic_height[i][j] may specify the resolution of the j-th sub-picture associated with the i-th source content region.

[0120] In various implementations, properties of each sub-picture or sub-picture group, such as bit depth, color subsampling, coding profile, and / or coding level, may be included in a sub-picture properties SEI message.

[0121] In various embodiments, the sub-picture attributes SEI message may indicate the number of sub-pictures associated with the same source content region in multiple representations or layers. The decoder may determine the corresponding reference sub-pictures available in the previous picture and derive the reference sub-picture positions and sizes from the active PPS to perform ARC. The disclosed SEI message may explicitly signal the position and size of each sub-picture, for example, to simplify the derivation so that the decoder does not have to parse the parameter set or SEI message for each representation or layer. The syntax elements of the disclosed sub-picture attributes SEI message may also be carried by a cross-layer parameter set such as a video parameter set (VPS) or a decoder parameter set (DPS).

[0122] In various embodiments, the middle box may rely on the disclosed SEI messages. For example, the middle box may extract a sub-picture that matches the viewport based on the SEI message. The middle box may use the extracted sub-picture to form an ARC picture, for example to reduce the frame size (e.g., the number of bits required to encode and decode the frame). The client may rely on the proposed SEI messages. For example, the client may identify an ARC sub-picture and an associated reference sub-picture based on the SEI message. The client may align the coordinates between the ARC sub-picture and the reference sub-picture for appropriate motion compensation processes.

[0123] Figure 7 An example of a sub-picture based ARC mechanism 700 is shown. Figure 7, each sub-picture in a cubemap projection format may be encoded and decoded into two resolutions. Sub-pictures that match the viewport may be extracted from the high-resolution representation. The remaining sub-pictures may be extracted from the low-resolution representation. In various embodiments, the sub-picture attribute SEI message may indicate those sub-pictures associated with the same source content region. For example, tile group 0 and tile group 6 both cover the left, and tile group 1 and tile group 7 both cover the front, but at different resolutions and with different representations.

[0124] In the case where the extractor extracts non-IRAP high-resolution sub-pictures to match viewport changes, the extractor may signal the occurrence of ARC using a sub-picture attribute SEI message. The decoder may determine that both the right sub-picture and the front sub-picture are ARC sub-pictures because sub-picture No. 2 (right) and sub-picture No. 7 (front) are not available in the previous picture. In order to reconstruct the ARC sub-pictures, the decoder may parse the SEI message and may identify those sub-pictures that are associated with the same area as tile group No. 2 and tile group No. 7. For example, tile group No. 1 may be associated with the same content area as tile group No. 7 and may be available in a previously decoded picture. Tile group No. 8 may be associated with the same content area as tile group No. 2 and may be available in a previously decoded picture. Based on the sub-picture position and size derived from the PPS, the decoder may scale down the decoded sub-picture No. 1 and scale up the decoded sub-picture No. 8 for ARC motion compensation. The motion vector for each sample of sub-picture No. 2 may (e.g., should) be shifted by the offset between the coordinates of sub-picture No. 2 and the coordinates of sub-picture No. 8, for example, as Figure 7 The (dMVx, dMVy) labeled in . For example, the offset may be in units of one-sixteenth sample spacing relative to the luma sampling grid. The same motion vector shift may also be applied to each sample of tile group 7.

[0125] In one embodiment, all sub-pictures associated with the same source content region may be assigned to a sub-picture group, and each sub-picture group may have its own unique ID. The sub-picture group ID and its attributes such as region position and size may be carried in the tile group header, or in the PPS tile group layout syntax field.

[0126] In one embodiment, SEI attribute messages may signal the attributes of the current sub-picture and its associated reference sub-pictures during ARC, including identifiers, coordinates, sub-picture size, bit depth, chroma subsampling, projection format, region-wise packing, etc. Table 2 provides an example of an ARC sub-picture attribute SEI message.

[0127] Table 2 – Example of ARC sub-picture attributes SEI message

[0128]

[0129] In Table 2, num_arc_subpics_minus1 plus 1 may specify the number of sub-pictures associated with the adaptive resolution change.

[0130] In Table 2, arc_subpic_id[i] and arc_subpic_id[i] may specify identifiers of the i-th ARC sub-picture and the associated reference sub-picture.

[0131] In Table 2, arc_subpic_coordinate[i] and arc_subpic_size[i] may specify the coordinates (eg, center or top-left sample position) and size of the i-th ARC sub-picture.

[0132] In Table 2, reference_subpic_coordinate[i] and reference_subpic_size[i] may specify the coordinates (eg, center or top-left sample position) and size of the reference sub-picture of the i-th ARC sub-picture.

[0133] In various implementations, the ARC sub-picture SEI message may include information indicating the format of the ARC sub-picture and associated reference sub-pictures, including bit depth, color subsampling, coding profile, and / or coding level.

[0134] In various embodiments, the ARC sub-picture SEI message may include information indicating the coded sub-picture after ARC and the corresponding reference sub-picture before ARC. The client may scale or transform (e.g., mirror, scale, or rotate) the corresponding reference picture for motion compensation based on such instructions. The message may (e.g., typically) be generated by an intermediate frame or extractor that extracts and repositions the sub-picture bitstream. The message may (e.g., should) be attached to the ARC picture. Based on such a message, the client or decoder may identify the ARC sub-picture and the associated reference sub-picture, and align the coordinates between the two sub-pictures for motion compensation.

[0135] Recommended ARC Transformation SEI Message

[0136] In various embodiments, ARC can use scaled reconstructed pictures as reference pictures, for example, to avoid high bit rate IDR or IRAP pictures and / or maintain acceptable decoded picture quality. Multiple versions of coded video of the same content may exist. ARC transition performance including reconstructed picture quality and error propagation may depend on scaling filter design, reference pictures, and temporal layers.

[0137] Figure 8A temporal scalability mechanism 800 is shown, where an ARC transition on POC No. 4 may not cause error propagation. Based on the correlation between sub-pictures in different versions and codec structures, the encoder may know the optimal ARC transition point. For example, the encoder may simulate ARC in each IRAP interval or picture group (DSP) interval to determine the ARC transition point.

[0138] Fig. 9 An example of determining the optimal ARC transition point is shown. In addition to conventional temporal inter-frame prediction and motion compensation, the encoder can simulate reconstructed pictures by referencing scaled pictures from different representations, calculate PSNR, and mark the picture that achieves the highest peak signal-to-noise ratio (PSNR) as the recommended ARC transition point.

[0139] In one embodiment, to determine the ARC transition point (eg, POC No. 3), the mechanism 900 may periodically perform inter-layer prediction as specified in SHVC between two representations.When performing ARC, additional inter-layer codec data may be delivered to the end user.

[0140] Fig.10 An exemplary mechanism 1000 is shown in which high-resolution representation pictures are periodically predicted from scaled low-resolution IRAP pictures. These pictures are marked as ARC transition pictures. The transition from low-resolution to high-resolution representation can occur only at these ARC transition points, while the transition from high-resolution to low-resolution representation can occur at any low-resolution IRAP picture because the size of low-resolution IRAP pictures is much smaller than that of high-resolution IRAP pictures. When performing ARC, low-resolution IRAP pictures can be carried in the bitstream as inter-layer reference pictures for high-resolution pictures.

[0141] Still see Fig.10 , bitstream 1 may be a temporally predicted low-resolution codec stream with periodic IRAP pictures, and bitstream 2 may be a temporally predicted high-resolution codec stream with ARC pictures inter-layer predicted from low-resolution IRAP pictures. The ARC transition bitstream may include a low-resolution reference picture (POC No. 4), an inter-layer predicted ARC picture (POC No. 4), and a subsequent temporally predicted high-resolution picture.

[0142] In various embodiments, the recommended ARC transition SEI message may be included in the sub-picture attribute SEI message, or used or transmitted separately. The recommended ARC transition SEI message may include the POC value of the high-resolution representation, the POC value of the corresponding lower-layer reference picture with a representation ID (such as a layer ID or a tile group ID), the scaling filter coefficients, and the prediction method (e.g., temporal prediction or inter-layer prediction).

[0143] Table 3 provides an exemplary syntax of a recommended ARC transition SEI message. The message identifies the recommended ARC sub-picture with its layer ID, POC value, and sub-picture ID. The message also indicates the multiple sub-pictures recommended to switch from. Scaling filters can support customized scaling filters to improve ARC quality.

[0144] Table 3 – Example of recommended ARC transition SEI message

[0145]

[0146] In Table 3, arc_layer_id may specify the identifier of the layer with which the ARC sub-picture is associated.

[0147] In Table 3, arc_poc_lsb may specify the POC LSB value of the ARC sub-picture.

[0148] In Table 3, arc_subpic_id may specify an identifier of an ARC sub-picture.

[0149] In Table 3, num_ref_subpic_minus1 plus one may specify the number of recommended sub-pictures switched from the ARC period.

[0150] In Table 3, reference_subpic_id[i] may specify the i-th sub-picture switched from the ARC period.

[0151] In Table 3, scaling_filter may be a structure containing recommended scaling filter coefficients.

[0152] In various embodiments, the SEI message may provide multiple recommended ARC transition points and prioritize these points. A priority indicator may be signaled in the SEI message to indicate the priority of the sub-picture for performing ARC. A higher priority indication may achieve better ARC performance on the associated sub-picture.

[0153] ARC toggle indicator

[0154] Indicators can be used to signal PPS or sub-picture related parameter sets in SEI messages to indicate to the decoder that ARC has occurred. Indicators can carry parameters such as layer IDs and POC numbers. Since some pictures after the ARC switching point will use scaled pictures as references, motion compensation errors may impair the performance of those codec tools that rely on accurate temporal information and reference samples. For example, bidirectional optical flow (BDOF) requires the use of the difference and spatial gradient between two temporal predictions from forward and backward predictions to derive the delta motion vector for each 4×4 sub-block based on the temporal prediction signal. After the spatial scaling of the reference picture after the ARC switching point, both the temporal difference and the spatial gradient will change, and the derived results will change greatly. Decoder-side motion vector refinement (DMVR) uses two temporal predictions from two reference pictures to derive the delta motion vector for each sub-block (e.g., 16×16) within the codec unit. If the temporal reference picture is scaled, the derived results will be changed. Decoder-side derived related codec techniques such as BDOF and DMVR should be disabled at all subsequent pictures before the ARC picture and the next IRAP picture in the decoding order. In various implementations, these codec techniques may be disabled based on indicators signaled in ARC-related SEI messages, or they may be adaptively disabled based on the scaling ratio. If the scaling ratio is close to 1, they may be enabled; otherwise, if the scaling ratio is far from 1, they may be disabled. Disabling those codec tools may further reduce decoder workload and power consumption without affecting codec performance.

[0155] Temporal motion vector prediction (TMVP) and sub-block temporal motion vector prediction (SbTMVP) may also be affected by motion vector scaling due to resolution changes, and motion vector errors may be propagated within a picture due to motion vector prediction. Due to motion compensation, the error of the motion vector has a great impact on the quality of the picture. In various embodiments, the encoder may disable these decoder-side derived related coding techniques TMVP and SbTMVP when encoding those pictures (such as those inter-frame pictures after the ARC switching point that may be affected by reference picture scaling and motion vector scaling), so that the picture quality will not be affected too much after the ARC switching.

[0156] In various embodiments, an indication or flag (e.g., a constraint flag) may be used to indicate whether one or more codec tools (e.g., a predetermined codec tool) are disabled at an ARC transition point. In one example, the decoder may detect a constraint flag set in the SPS, and the constraint flag indicates that the constraints of one or more codec tools apply to the entire coded video sequence (CVS). For example, layer-based spatial scalability may allow each high-resolution enhancement picture to be predicted from a low-resolution (e.g., base layer) picture, and the constraints of one or more codec tools are applied to each enhancement layer (e.g., a layer other than the base layer as the first layer in the bitstream) frame. For a single-layer sequence, the encoder may set a constraint flag on some or each determined frame for ARC transition. The decoder may detect the flag at the slice level and perform ARC on the associated frame (e.g., a frame containing a slice).

[0157] Table 4 provides an exemplary syntax of an SPS, including a constraint flag sps_arc_constraint_flag, which is signaled in the SPS.

[0158] Table 4 – Example of SPS syntax

[0159]

[0160]

[0161] In Table 4, sps_arc_constraint_flag is a constraint flag. When the value of sps_arc_constraint_flag is equal to 1, it indicates or specifies that the codec tools TMVP, DMVR and / or BDOF (e.g., slice_temporal_mvp_enabled_flag, bdofFlag and / or dmvrFlag having a value equal to 0) should be disabled for the CVS associated with the SPS. When sps_arc_constraint_flag has a value equal to 0, the flag indicates or specifies that it does not impose constraints.

[0162] Table 5 provides an exemplary syntax of a slice header. The constraint flag slice_arc_constraint_flag is signaled in the slice header.

[0163] Table 5 – Example of slice header syntax

[0164]

[0165] In Table 5, slice_arc_constraint_flag is a constraint flag. When the value of slice_arc_constraint_flag is equal to 1, it indicates or specifies that the codec tools TMVP, DMVR and / or BDOF should be disabled for the associated slice (e.g., slice_temporal_mvp_enabled_flag, bdofFlag and / or dmvrFlag of the associated slice having a value equal to 0). When slice_arc_constraint_flag has a value equal to 0, the flag indicates or specifies that it does not impose constraints. When slice_arc_constraint_flag is not present, it is inferred to have a value equal to 1.

[0166] In various embodiments, the constraints of the codec tool (e.g., as signaled using one or more constraint flags described above) may indicate (or provide reference to) a known or fixed set of tools to be disabled after an ARC transition. For example, the constraints of the codec tool may disable BDOF and DMVR after an ARC transition and / or until the next IRAP picture. In another embodiment, additional signaling (e.g., one or more additional flags) may be provided in the syntax to adaptively specify which codec tools are to be disabled when the constraints of the codec tool are in effect. For example, a first flag may specify whether BDOF is to be disabled by the constraints of the codec tool, a second flag may specify whether DMVR is to be disabled by the constraints of the codec tool, and one or more additional flags may specify whether other tools are to be disabled by the constraints of the codec tool.

[0167] In various embodiments, Table 6 provides an exemplary syntax for a VPS, where the full layer independent flag is used to indicate that each layer is independently encoded and decoded, so that dependencies between layers do not need to be specified.

[0168] Table 6 – Examples of VPS syntax

[0169]

[0170] In Table 6, vps_all_layers_independent_flag equal to 1 may specify that each layer (specified by the VPS) is an independent layer, and vps_all_layers_independent_flag equal to 0 may specify that one or more layers (specified by the VPS) may not be independent layers. When vps_all_layers_independent_flag is set to 1, layer dependency signaling (e.g., direct_dependency_flag) does not have to be signaled.

[0171] In various embodiments, methods and apparatus for picture and / or video encoding and decoding in a communication system are provided. For example, the method may include selectively including sub-picture attribute information in an SEI message for use with adaptive switching of viewports, and generating / transmitting an SEI message. The method may also include identifying a set of sub-pictures associated with the viewport, and identifying sub-picture attribute information associated with the identified set of sub-pictures. In one example, selectively including sub-picture attribute information in an SEI message may include generating an SEI message including the identified sub-picture attribute information.

[0172] In various embodiments, the sub-picture attribute information indicates and / or includes any of the following: one or more layer identifiers (IDs), one or more tile group IDs, coordinates of each sub-picture, the position and format of each sub-picture in a sub-picture set, mapping information of each sub-picture to be mapped to the spherical coordinate space of the picture, bit depth of one or more sub-pictures, color subsampling information, encoding profile, and encoding level.

[0173] In various embodiments, the method may include generating a syntax to indicate sub-picture attribute information in a SEI message.

[0174] In various embodiments, a sub-picture set may be a set of tile groups associated with a picture.

[0175] In various embodiments, the method may include selectively including sub-picture attribute information in a SEI message for use with ARC in conjunction with adaptive switching of viewports, and generating / transmitting the SEI message. In one example, the sub-picture attribute information may include information indicating either a low-resolution sub-picture or a high-resolution sub-picture, to which ARC may be applied for adaptive switching of viewports. In one example, the low-resolution sub-picture and the high-resolution sub-picture are associated with the same source content region for the viewport.

[0176] In various implementations, the SEI message may be transmitted after a frame including a first IRAP picture and before a next frame including a second IRAP picture.

[0177] In various embodiments, any of the methods discussed herein may include identifying a sub-picture set associated with a picture that can be used for ARC, and the sub-picture set includes a low-resolution sub-picture and a high-resolution sub-picture; and identifying sub-picture attribute information associated with the identified sub-picture set. In one example, selectively including the sub-picture attribute information in the SEI message may include generating an SEI message including the identified sub-picture attribute information.

[0178] In various embodiments, a method may include receiving a SEI message including sub-picture attribute information for use with adaptive switching of viewports; and performing ARC based on the sub-picture attribute information.

[0179] In various implementations, the sub-picture attribute information may include information indicating any one of a low-resolution sub-picture and a high-resolution sub-picture to which ARC may be applied for adaptive switching of viewports.

[0180] In various implementations, ARC may be performed at least in part by resizing an indicated low-resolution sub-picture to a high-resolution sub-picture and / or resizing an indicated high-resolution sub-picture to a low-resolution sub-picture.

[0181] In various embodiments, performing ARC may include performing ARC in conjunction with adaptive switching of viewports based on sub-picture attribute information.

[0182] In various implementations, ARC may be performed without introducing an IRAP picture frame.

[0183] In various implementations, ARC may be performed at least in part by resizing indicated low-resolution sub-pictures to high-resolution sub-pictures and resizing indicated high-resolution sub-pictures to low-resolution sub-pictures.

[0184] In various implementations, any of the methods discussed herein may include decoding a received SEI message.

[0185] In various embodiments, performing ARC may include upscaling the low-resolution sub-picture based on the sub-picture attribute information.

[0186] In various embodiments, performing ARC may include scaling down the high-resolution sub-picture based on the sub-picture attribute information.

[0187] In various implementations, performing ARC may include aligning coordinates between a sub-picture set and a corresponding reference sub-picture set based on the sub-picture attribute information.

[0188] In various embodiments, performing ARC may include determining a position and format of a sub-picture based on sub-picture attribute information, determining a resolution of a corresponding reference sub-picture, scaling the resolution of the sub-picture based on the determined position and format and the resolution of the reference sub-picture, and mapping the scaled sub-picture to corresponding coordinates of the picture.

[0189] In various implementations, a sub-picture may include a group of tiles.

[0190] In various embodiments, any of the methods discussed herein may include determining whether a corresponding reference sub-picture is collocated with a sub-picture based on sub-picture property information.

[0191] In various implementations, the SEI message may include syntax with sub-picture attribute information.

[0192] In various implementations, syntax may be carried by cross-layer parameter sets.

[0193] In various embodiments, the sub-picture attribute information may indicate the coordinates of each repositioned sub-picture relative to the consistent cropping window.

[0194] In various embodiments, any of the methods discussed herein may include determining corresponding reference sub-pictures available in one or more previous pictures, and deriving the location and format of the corresponding reference sub-pictures based on the sub-picture attribute information.

[0195] In various embodiments, the sub-picture attribute information may indicate attributes of a set of sub-pictures including one or more low-resolution sub-pictures and one or more high-resolution sub-pictures.

[0196] In various embodiments, the method may include identifying an ARC switching point, wherein one or more pictures following the ARC switching point use the scaled picture as a reference; and sending an indicator indicating the ARC switching point.

[0197] In various implementations, the indicator is transmitted in any of: a SEI message, a picture parameter set (PPS), and a sub-picture related parameter set.

[0198] In various implementations, the indicator comprises a parameter, wherein the parameter comprises any of: a layer ID and a picture order count (POC) value.

[0199] In various embodiments, the method may include receiving an indicator indicating an ARC switching point, and identifying the ARC switching point and a scaled picture. In various embodiments, one or more pictures received after the ARC switching point may use the scaled picture as a reference.

[0200] In various embodiments, the method may include identifying a sub-picture set associated with a picture that is available or recommended for performing ARC, and generating / sending a SEI message indicating one or more parameters of the sub-picture set.

[0201] In various implementations, the one or more parameters discussed above may include any of: a POC value of a high-resolution representation of a picture, a POC value of a corresponding lower layer reference picture with a representation ID, one or more scaling filter coefficients, and a prediction method.

[0202] In various embodiments, the representation ID may be a layer ID or a tile group ID.

[0203] In various implementations, the prediction method may include any of the following: temporal prediction and inter-layer prediction.

[0204] In various implementations, the syntax may indicate information of a sub-picture in a sub-picture set, wherein the information includes any of: a layer ID, a POC value, and a sub-picture ID associated with the sub-picture.

[0205] In various embodiments, the method may include receiving a SEI message associated with a picture, and identifying a set of sub-pictures recommended for performing ARC based on one or more parameters indicated in the SEI message, and selecting one or more sub-pictures from the set of sub-pictures to perform ARC.

[0206] In various implementations, the SEI message may include a priority indicator indicating the priority of a corresponding sub-picture in the set of sub-pictures.

[0207] In various embodiments, the priority discussed herein may be a high priority indicating better ARC performance on the corresponding sub-picture.

[0208] In various embodiments, an apparatus may include one or more processors, encoders, decoders, transmitters, receivers, and / or memories that implement or perform any of the methods discussed herein.

[0209] In various embodiments, the middle block may include one or more processors, encoders, decoders, transmitters, receivers, and / or memories that implement or perform any of the methods discussed herein.

[0210] Each of the following references is incorporated herein by reference: [1] JCTVC-F158, “Resolution switching for coding efficiency and resilience,” July 2011; [2] JVET-M0135, “On adaptive resolution change for VVC,” January 2019; [3] JVET-M0259, “Use cases and proposed design choices or adaptive resolution changing,” January 2019; [4] JVET-M0261, “AHG12: On grouping of tiles,” January 2019; and [5] U.S. Provisional Patent Application No. 62 / 775,130.

[0211] in conclusion

[0212] Although features and elements are described above in particular combinations, it will be understood by one of ordinary skill in the art that each feature or element may be used alone or in any combination with other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of non-transitory computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media (such as built-in hard disks and removable disks), magneto-optical media, and optical media (such as CD-ROM disks and digital versatile disks (DVDs)). A processor associated with software may be used to implement a radio frequency transceiver for a WTRU 102, UE, terminal, base station, RNC, or any host computer.

[0213] In addition, in the above-mentioned embodiments, processing platforms, computing systems, controllers, and other devices including processors are indicated. These devices may include at least one central processing unit ("CPU") and memory. According to the practice of those skilled in the art of computer programming, references to the symbolic representations of actions and operations or instructions may be performed by various CPUs and memories. Such actions and operations or instructions may be considered to be "executed," "computer executed," or "CPU executed."

[0214] Those of ordinary skill in the art will appreciate that the actions and symbolic representations of operations or instructions include manipulation of electrical signals by the CPU. The electrical system represents data bits, which may result in the ultimate transformation of electrical signals or reduction of electrical signals and the retention of data bits at memory locations in the memory system, thereby reconfiguring or otherwise changing the operation of the CPU and performing other processing of signals. The memory location that holds the data bits is a physical location that has specific electrical, magnetic, optical, or organic properties that correspond to or represent the data bits. It should be understood that the representative embodiments are not limited to the above-mentioned platforms or CPUs, and other platforms and CPUs may also support the provided methods.

[0215] The data bits may also be maintained on computer-readable media, including magnetic disks, optical disks, and any other volatile (e.g., random access memory ("RAM")) or non-volatile (e.g., read-only memory ("ROM")) mass storage system readable by the CPU. The computer-readable media may include cooperating or interconnected computer-readable media that reside solely on the processing system or distributed across multiple interconnected processing systems that may be local or remote relative to the processing system. It should be understood that the representative embodiments are not limited to the above-described memories, and other platforms and memories may also support the described methods.

[0216] In an exemplary embodiment, any of the operations, processes, etc. described herein may be implemented as computer-readable instructions stored on a computer-readable medium. The computer-readable instructions may be executed by a processor of a mobile unit, a network element, and / or any other computing device.

[0217] There is little difference between hardware implementation and software implementation of various aspects of the system. The use of hardware or software is usually (for example but not always, because in some contexts, the choice between hardware and software may become important) a design choice that represents a trade-off between cost and efficiency. There may be various media (for example, hardware, software and / or firmware) that can affect the processes and / or systems and / or other technologies described herein, and the preferred media may vary with the context of the deployment process and / or system and / or other technologies. For example, if the implementer determines that speed and accuracy are most important, the implementer may choose a medium that is primarily hardware and / or firmware. If flexibility is most important, the implementer may choose a specific implementation that is primarily software. Alternatively, the implementer may choose some combination of hardware, software and / or firmware.

[0218] The above detailed description has listed various embodiments of the device and / or process by using block diagrams, flow charts and / or examples. Where such block diagrams, flow charts and / or examples contain one or more functions and / or operations, it will be understood by those skilled in the art that each function and / or operation within such block diagrams, flow charts or examples may be implemented individually and / or collectively by a wide range of hardware, software, firmware or nearly any combination thereof. Suitable processors include (shown by way of example) general purpose processors, special purpose processors, conventional processors, digital signal processors (DSPs), multiple microprocessors, one or more microprocessors associated with a DSP core, controllers, microcontrollers, application specific integrated circuits (ASICs), application specific standard products (ASSPs), field programmable gate arrays (FPGAs) circuits, any other type of integrated circuit (IC) and / or state machines.

[0219] Although features and elements are provided above in specific combinations, it will be understood by those of ordinary skill in the art that each feature or element may be used alone or in any combination with other features and elements. The present disclosure is not limited to the specific embodiments described in this patent application, which are intended to be examples of various aspects. Many modifications and variations can be made without departing from the essence and scope of the present invention, as they will be obvious to those skilled in the art. Unless explicitly provided as such, any element, action or description used in the present application specification should not be understood as being essential or necessary to the present invention. According to the foregoing description, in addition to those listed herein, functionally equivalent methods and devices within the scope of the present disclosure will be obvious to those skilled in the art. Such modifications and variations are intended to fall within the scope of the appended claims. The present disclosure is limited only to the terms of the appended claims and the full range of equivalent forms of such entitled claims. It should be understood that the present disclosure is not limited to a specific method or system.

[0220] It should also be understood that the terms used herein are for the purpose of describing specific embodiments only and are not intended to be limiting. As used herein, the term "station" and its abbreviation "STA", "user equipment" and its abbreviation "UE" when referred to herein may mean: (i) a wireless transmit and / or receive unit (WTRU), such as described below; (ii) any of several embodiments of the WTRU, such as described below; (iii) a device with wireless functionality and / or with wired functionality (e.g., tetherable) configured with (in particular) some or all of the structures and functions of the WTRU, such as described below; (iii) a device with wireless functionality and / or with wired functionality configured with structures and functions that are less than all of the structures and functions of the WTRU, such as described below; or (iv), etc. The following relative to Figures 1A to 1DDetails are provided for an exemplary WTRU that may represent (or be interchangeable with) any UE or mobile device described herein.

[0221] In certain representative embodiments, several portions of the subject matter described herein may be implemented via application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), and / or other integrated formats. However, those skilled in the art will recognize that some aspects of the embodiments disclosed herein may be equivalently implemented in whole or in part in an integrated circuit as one or more computer programs running on one or more computers (e.g., one or more programs running on one or more computer systems), one or more programs running on one or more processors (e.g., one or more programs running on one or more microprocessors), firmware, or nearly any combination thereof, and that designing circuits and / or writing code for software and / or firmware will be well within the technical scope of those skilled in the art in light of the present disclosure. In addition, those skilled in the art will appreciate that the mechanisms of the subject matter described herein may be distributed as program products in a variety of forms, and that the exemplary embodiments of the subject matter described herein apply regardless of the specific type of signal-bearing medium used to actually perform the distribution. Examples of signal-bearing media include, but are not limited to, the following: recordable type media (such as floppy disks, hard drives, CDs, DVDs, digital tapes, computer memory, etc.); and transmission type media (such as digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, etc.)).

[0222] The subject matter described herein sometimes shows different parts contained in or connected to different other parts.It should be understood that such depicted architectures are merely examples, and in fact many other architectures that achieve the same function can be realized.In a conceptual sense, any arrangement of parts that achieve the same function is effectively "associated" so that the desired function can be achieved.Therefore, any two parts combined herein to achieve a specific function can be considered to be "associated" with each other so that the desired function is achieved, regardless of the architecture or intermediate parts.Similarly, any two parts so associated can also be considered to be "operably connected" or "operably coupled" to each other to achieve the desired function, and any two parts that can be so associated can also be considered to be "operably coupleable" to each other to achieve the desired function.Specific examples of operable coupling include, but are not limited to, parts that can physically cooperate and / or physically interact and / or parts that can wirelessly interact and / or wirelessly interact and / or parts that can logically interact and / or parts that can logically interact.

[0223] With respect to substantially any plural and / or singular terms used herein, those skilled in the art may translate from the plural to the singular and / or from the singular to the plural as appropriate, as appropriate to the context and / or application. For clarity, various singular / plural arrangements may be explicitly listed herein.

[0224] It will be understood by those skilled in the art that, in general, the terms used herein, especially in the appended claims (e.g., the bodies of the appended claims), are generally intended to be "open" terms (e.g., the term "including" should be interpreted as "including but not limited to", the term "having" should be interpreted as "having at least", the term "comprising" should be interpreted as "including but not limited to", etc.). It will also be understood by those skilled in the art that if a specific number of an introduced claim recitation object is intended, such intent will be explicitly stated in the claim, and in the absence of such a recitation object, there is no such intent. For example, where only one item is intended, the term "single" or similar language may be used. To aid understanding, the following appended claims and / or the description herein may contain the use of the introductory phrases "at least one" and "one or more" to introduce claim recitation objects. However, the use of such phrases should not be understood to imply that any particular claim containing such introduced claim recitation object is limited to an embodiment containing only one such recitation object by the indefinite article "a" or "an" to introduce the claim recitation object. This is true even when the same claim includes the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an" (e.g., "a" and / or "an" should be interpreted as meaning "at least one" or "one or more"). The same applies to the use of definite articles to introduce claim recitations. In addition, even if a specific number of an introduced claim recitation is explicitly recited, one skilled in the art will recognize that such recitation should be interpreted to mean at least the recited number (e.g., the bare recitation of "two recitations" means at least two recitations, or two or more recitations, in the absence of other modifiers).

[0225] Additionally, in those instances where a convention similar to “at least one of A, B, and C, etc.” is used, generally speaking, such constructions are meant to be understood by those skilled in the art (e.g., “a system having at least one of A, B, and C” would include, but is not limited to, systems having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). In those instances where a convention similar to “at least one of A, B, or C, etc.” is used, generally speaking, such constructions are meant to be understood by those skilled in the art (e.g., “a system having at least one of A, B, or C” would include, but is not limited to, systems having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). Those skilled in the art should also understand that, in fact, any separate word and / or phrase presenting two or more alternative terms, whether in the specification, claims, or drawings, should be understood to contemplate the possibility of including one of the terms, any of the terms, or both of the terms. For example, the phrase "A or B" will be understood to include the possibilities of "A" or "B" or "A and B." Additionally, as used herein, the term "any of" followed by a listing of multiple items and / or multiple categories of items is intended to include "any of," "any combination," "any multiples," and / or "any combination of multiples" of the items and / or categories of items, either alone or in combination with other items and / or other categories of items. Additionally, as used herein, the terms "group" or "grouping" are intended to include any number of items, including zero. Additionally, as used herein, the term "quantity" is intended to include any quantity, including zero.

[0226] In addition, where features or aspects of the disclosure are described in terms of Markush groups, those skilled in the art will recognize that the disclosure is also described in terms of any individual member or subgroup of members of the Markush group.

[0227] As will be understood by those skilled in the art, for any and all purposes (such as for providing a written description), all ranges disclosed herein also encompass any and all possible subranges and combinations of their subranges. Any listed range can be easily identified as fully describing and enabling the same range to be divided into at least equal halves, thirds, quarters, fifths, tenths, etc. As a non-limiting example, each range discussed herein can be easily divided into a lower third, a middle third, and an upper third, etc. As will be understood by those skilled in the art, all languages ​​such as "up to", "at least", "greater than", "less than", etc. include the cited numbers and refer to ranges that can subsequently be divided into subranges as described above. Finally, as will be understood by those skilled in the art, ranges include each individual number. Thus, for example, a group having 1 to 3 units refers to a group having 1, 2, or 3 units. Similarly, a group having 1 to 5 units refers to a group having 1, 2, 3, 4, or 5 units, etc.

[0228] Furthermore, the claims should not be read as limited to the order or elements presented unless otherwise stated. Additionally, the use of the term "means for..." in any claim is intended to invoke 35 U.S.C. § 112, 6 or means-plus-function claim format, and any claim without the term "means for..." is not intended to be so.

[0229] A processor in association with software may be used to implement the use of a radio frequency transceiver in a wireless transmit receive unit (WTRU), user equipment (UE), terminal, base station, mobility management entity (MME) or evolved packet core (EPC) or any host. The WTRU may be used in conjunction with a module and may be implemented in hardware and / or software including: a software defined radio (SDR) and other components such as a camera, video camera module, videophone, speakerphone, vibration device, speaker, microphone, television transceiver, hands-free headset, keyboard, module, a frequency modulation (FM) radio unit, a near field communication (NFC) module, a liquid crystal display (LCD) display unit, an organic light emitting diode (OLED) display unit, a digital music player, a media player, a video game player module, an internet browser, and / or any wireless local area network (WLAN) or ultra-wideband (UWB) module.

[0230] Although the present invention has been described in terms of a communication system, it is contemplated that the system may be implemented in software on a microprocessor / general purpose computer (not shown). In certain embodiments, one or more of the functions of the various components may be implemented in software controlling a general purpose computer.

[0231] Furthermore, while the invention has been shown and described herein with reference to specific embodiments, the invention is not intended to be limited to the details shown. Rather, various modifications may be made to the details within the scope and range of equivalents of the claims without departing from the invention.

[0232] Throughout the disclosure, it will be appreciated by those skilled in the art that certain representative embodiments may be used alternatively or in combination with other representative embodiments.

[0233] Although features and elements are described above in particular combinations, it will be understood by those of ordinary skill in the art that each feature or element may be used alone or in any combination with other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of non-transitory computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media (such as built-in hard disks and removable disks), magneto-optical media, and optical media (such as CD-ROM disks and digital versatile disks (DVDs)). A processor associated with software may be used to implement a radio frequency transceiver for a WTRU, UE, terminal, base station, RNC, or any host computer.

[0234] In addition, in the above-mentioned embodiments, processing platforms, computing systems, controllers, and other devices including processors are indicated. These devices may include at least one central processing unit ("CPU") and memory. According to the practice of those skilled in the art of computer programming, references to the symbolic representations of actions and operations or instructions may be performed by various CPUs and memories. Such actions and operations or instructions may be considered to be "executed," "computer executed," or "CPU executed."

[0235] Those of ordinary skill in the art will appreciate that the actions and symbolically represented operations or instructions include manipulation of electrical signals by the CPU. The electrical system represents data bits that may result in the ultimate transformation of the electrical signal or reduction of the electrical signal and the retention of the data bits at memory locations in the memory system to reconfigure or otherwise change the operation of the CPU and perform other processing of the signal. The memory location that holds the data bit is a physical location having specific electrical, magnetic, optical, or organic properties that correspond to or represent the data bit.

[0236] The data bits may also be maintained on computer-readable media, including magnetic disks, optical disks, and any other volatile (e.g., random access memory ("RAM")) or non-volatile (e.g., read-only memory ("ROM")) mass storage system readable by the CPU. The computer-readable media may include cooperating or interconnected computer-readable media that reside solely on the processing system or distributed across multiple interconnected processing systems that may be local or remote relative to the processing system. It should be understood that the representative embodiments are not limited to the above-described memories, and other platforms and memories may also support the described methods.

[0237] Suitable processors include, by way of example, a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), an application specific standard product (ASSP), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), and / or state machine.

[0238] Although the present invention has been described in terms of a communication system, it is contemplated that the system may be implemented in software on a microprocessor / general purpose computer (not shown). In certain embodiments, one or more of the functions of the various components may be implemented in software controlling a general purpose computer.

[0239] Furthermore, while the invention has been shown and described herein with reference to specific embodiments, the invention is not intended to be limited to the details shown. Rather, various modifications may be made to the details within the scope and range of equivalents of the claims without departing from the invention.

Claims

1. A method, comprising: selectively including sub-picture attribute information in a supplemental enhancement information (SEI) message for use with adaptive switching of viewports; as well as The SEI message is included in the bitstream.

2. The method according to claim 1, further comprising: Identifying a set of sub-images associated with the viewport; as well as Identifying the sub-picture attribute information associated with the identified sub-picture set, wherein selectively including the sub-picture attribute information in the SEI message comprises: The SEI message including the identified sub-picture attribute information is generated.

3. The method of claim 1 , wherein the sub-picture attribute information indicates or includes any one of: one or more layer identifiers (IDs), one or more tile group IDs, coordinates of each sub-picture, a position and format of each sub-picture in a sub-picture set, mapping information of each sub-picture to be mapped onto a spherical coordinate space of the picture, a bit depth of one or more sub-pictures, color subsampling information, an encoding profile, or an encoding level.

4. The method according to claim 1, wherein selectively including the sub-picture attribute information in the SEI message comprises: Syntax is generated to indicate the sub-picture attribute information in the SEI message. The method of claim 2 , wherein the set of sub-pictures is a set of tile groups associated with the picture.

6. A method comprising: receiving a supplemental enhancement information (SEI) message associated with a picture; identifying a set of sub-pictures recommended to perform adaptive resolution change (ARC) based on one or more parameters indicated in the SEI message; as well as One or more sub-pictures are selected from the set of sub-pictures to perform the ARC.

7. The method of claim 6, wherein the one or more parameters include any of the following: a picture order count (POC) value of a high resolution representation of the picture, the POC value of the corresponding lower-resolution reference picture with representation ID, one or more scaling filter coefficients, or Prediction method. The method according to claim 7 , wherein the representation ID is a layer ID or a tile group ID.

9. The method according to claim 7, wherein the prediction method comprises any one of temporal prediction and inter-layer prediction.

10. The method of claim 6, wherein the SEI message includes syntax.

11. The method according to claim 10, wherein the syntax indicates information of a sub-picture of the sub-picture set, wherein: The information includes any one of a layer ID, a POC value, and a sub-picture ID associated with the sub-picture. 12 . The method of claim 6 , wherein the SEI message comprises a priority indicator indicating a priority of a corresponding sub-picture in the sub-picture set. 13 . The method of claim 12 , wherein the priority is a high priority, indicating that the ARC performance on the corresponding sub-picture is better.

14. An apparatus comprising one or more processors, the one or more processors being configured to: selectively including sub-picture attribute information in a supplemental enhancement information (SEI) message for use with adaptive switching of viewports; and The SEI message is included in the bitstream.

15. The apparatus of claim 14, wherein the one or more processors are further configured to: identifying a set of sub-pictures associated with the viewport; and Identifying the sub-picture attribute information associated with the identified sub-picture set, wherein selectively including the sub-picture attribute information in the SEI message comprises: The SEI message including the identified sub-picture attribute information is generated.

16. A device according to claim 14, wherein the sub-picture attribute information indicates or includes any of the following: one or more layer identifiers (IDs), one or more tile group IDs, coordinates of each sub-picture, the position and format of each sub-picture in a sub-picture set, mapping information of each sub-picture to be mapped to the spherical coordinate space of the picture, bit depth of one or more sub-pictures, color subsampling information, encoding profile, or encoding level. 17 . The apparatus of claim 14 , wherein the one or more processors are further configured to generate syntax to indicate the sub-picture property information in the SEI message.

18. The device of claim 15, wherein the set of sub-pictures is a set of tile groups associated with the picture.

19. An apparatus comprising one or more processors, the one or more processors being configured to: receiving a supplemental enhancement information (SEI) message associated with a picture; identifying a sub-picture set recommended to perform adaptive resolution change (ARC) based on one or more parameters indicated in the SEI message; and One or more sub-pictures are selected from the set of sub-pictures to perform the ARC.

20. The apparatus of claim 19, wherein the one or more parameters include any of the following: a picture order count (POC) value of a high resolution representation of the picture, the POC value of the corresponding lower-resolution reference picture with representation ID, one or more scaling filter coefficients, or Prediction method.

Images