DASH signaling for adaptive streaming of haptic media

Abstract

Some embodiments of a method may include encoding information describing a haptic experience, where the information describing the haptic experience includes one or more adaptation sets, where each of the one or more adaptation sets includes one or more representations corresponding to a haptic media track. Some embodiments of a method may include decoding information describing a haptic experience, where the information describing the haptic experience includes one or more adaptation sets, where each of the one or more adaptation sets includes one or more representations corresponding to a haptic media track.

Description

[0001] Cross-references to related applications This application claims the benefit of U.S. Patent Application No. 63 / 526,954, filed July 14, 2023, entitled “DASH SIGNALING FOR ADAPTIVESTREAMING OF HAPTICS MEDIA”, which is incorporated herein by reference in its entirety.

[0002] Merge by reference This application incorporates in its entirety the following application by reference: filed on October 19, 2022, entitled " Carriage of Coded Haptics Data in Media Containers U.S. Provisional Patent Application Serial No. 63 / 417,638 (“638 Application”). Background Technology

[0003] A haptic sequence is a set of data encoded for haptic and spatial positioning-based rendering, similar to how a video sequence is a set of encoded data for visual rendering. Haptic sequences encode temporal data, such as tracks associated with a haptic device. A haptic device can render different modalities of haptic and spatial positioning, such as vibration, force, position, velocity, or temperature. Summary of the Invention

[0004] The embodiments described herein include methods for video encoding and decoding (collectively, “decoding”).

[0005] Example methods according to some embodiments may include: encoding information describing a tactile experience, wherein the information describing the tactile experience includes one or more adapter sets, wherein each of the one or more adapter sets includes one or more representations corresponding to a tactile media track.

[0006] In some embodiments of the example method, each of one or more representations corresponds to the same time period.

[0007] In some embodiments of the example method, the information is encoded in a container file.

[0008] For some embodiments of the example method, one or more adaptation sets may include a primary tactile experience and a second tactile experience, wherein the second tactile experience corresponds to a first perceptual modality and a first channel.

[0009] For some embodiments of the example method, the adapter set corresponding to the primary haptic experience may include initialization data corresponding to the haptic decoder.

[0010] For some embodiments of the example method, the adaptation set corresponding to the second tactile experience may include one or more segments of the corresponding tactile track.

[0011] For some embodiments of the example method, the adaptation set corresponding to the second haptic experience can include information identifying one or more segments of a corresponding haptic track.

[0012] For some embodiments of the example method, the first channel corresponds to a first frequency band.

[0013] For some embodiments of the example method, the information describing the haptic experience can further include information identifying one or more adaptation sets.

[0014] For some embodiments of the example method, the information describing the haptic experience can further include: information describing at least one available avatar for the haptic experience; and configuration information for at least one perception in the haptic experience, wherein the configuration information can include information describing one or more parallel haptic experience tracks.

[0015] For some embodiments of the example method, the information can further include information describing one or more haptic experience tracks, and the information describing the haptic experience track can further include: information describing at least one available avatar for the haptic experience; and configuration information for at least one perception in the haptic experience.

[0016] An example method / apparatus according to some embodiments can include: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to: encode information describing a haptic experience, wherein the information describing the haptic experience includes one or more adaptation sets, and wherein each of the one or more adaptation sets includes one or more representations corresponding to a haptic media track.

[0017] An additional example method according to some embodiments can include: decoding information describing a haptic experience, wherein the information describing the haptic experience includes one or more adaptation sets, and wherein each of the one or more adaptation sets includes one or more representations corresponding to a haptic media track.

[0018] For some embodiments of the additional example method, each of the one or more representations corresponds to a same time period.

[0019] For some embodiments of the additional example method, the encoding of the information encodes the information in a container file.

[0020] For some embodiments of the additional example method, the one or more adaptation sets can include a primary haptic experience and a second haptic experience, and the second haptic experience corresponds to a first perception modality and a first channel.

[0021] For some embodiments of the additional example method, the adaptation set corresponding to the primary haptic experience can include initialization data corresponding to a haptic decoder.

[0022] For some embodiments of the additional example method, the adaptation set corresponding to the second haptic experience can include one or more segments of a corresponding haptic track.

[0023] For some embodiments of the additional example method, the adaptation set corresponding to the second haptic experience can include information identifying one or more segments of a corresponding haptic track.

[0024] Some embodiments of the additional example method can further include concatenating the initialization data with information from one or more adaptation sets identifying one or more segments to generate a bitstream.

[0025] Some embodiments of the additional example method can further include rendering the bitstream in a haptic experience environment.

[0026] For some embodiments of the additional example method, the first channel corresponds to a first frequency band.

[0027] For some embodiments of the additional example method, the information describing the haptic experience can further include information identifying one or more adaptation sets.

[0028] For some embodiments of the additional example method, the information describing the haptic experience can further include: information describing at least one available avatar for the haptic experience; and configuration information for at least one perception in the haptic experience, and the configuration information can include information describing one or more parallel haptic experience tracks.

[0029] For some embodiments of the additional example method, the information can further include information describing one or more haptic experience tracks, and the information describing the haptic experience track can further include: information describing at least one available avatar for the haptic experience; and configuration information for at least one perception in the haptic experience.

[0030] For some embodiments of the example method, the information describing the haptic experience and / or the encoded information describing the haptic experience is encoded according to the ISO Base Media File Format (ISOBMFF).

[0031] For some embodiments of the example method, the information describing the haptic experience and / or the encoded information describing the haptic experience is included in and streamed in a media file such as an MPEG-DASH (Dynamic Adaptive Streaming over HTTP) Media Presentation Descriptor (MPD) file.

[0032] For some embodiments of the example method, the information describing the haptic experience and / or the encoded information describing the haptic experience uses, is part of, or conforms to one or more of the MPEG standards ISO / IEC 23090-32 and ISO / IEC 23090-31.

[0033] An example method / apparatus according to some embodiments can comprise: a processor; and a non-transitory computer readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to: decode information describing a haptic experience, wherein the information describing the haptic experience comprises one or more adaptation sets, and wherein each of the one or more adaptation sets comprises one or more representations corresponding to a haptic media track.

[0034] An example apparatus according to some embodiments can comprise at least one processor configured to perform any of the methods listed above.

[0035] An example apparatus according to some embodiments can comprise a computer readable medium storing instructions for causing one or more processors to perform any of the methods listed above.

[0036] An example apparatus according to some embodiments can comprise at least one processor and at least one non-transitory computer readable medium storing instructions for causing the at least one processor to perform any of the methods listed above.

[0037] An example apparatus according to some embodiments can comprise a computer readable medium storing a scene description file encoding generated according to any of the methods listed above.

[0038] An example signal according to some embodiments can comprise a scene description file generated according to any of the methods listed above.

[0039] In additional embodiments, encoder and decoder apparatus are provided to perform the methods described herein. The encoder or decoder apparatus can comprise a processor configured to perform the methods described herein. The apparatus can comprise a computer readable medium (e.g., a non-transitory medium) storing instructions for performing the methods described herein. In some embodiments, the computer readable medium (e.g., a non-transitory medium) stores video encoded using any of the methods described herein.

[0040] One or more embodiments also provide a computer-readable storage medium having instructions stored thereon for performing bi-directional optical flow, encoding or decoding video data according to any of the above methods. One or more embodiments also provide a computer-readable storage medium having a bitstream generated according to the above methods stored thereon. One or more embodiments also provide a method and apparatus for transmitting a bitstream generated according to the above methods. One or more embodiments also provide a computer program product comprising instructions for performing any of the described methods. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1A is a system diagram illustrating an example communication system, in accordance with some embodiments.

[0042] Figure 1B is a system diagram illustrating a set of example interfaces for a system, in accordance with some embodiments. Figure 1A is a system diagram illustrating an example wireless transmit / receive unit (WTRU) that can be used within the illustrated communication system, in accordance with some embodiments.

[0043] Figure 1C is a system diagram illustrating a set of example interfaces for a system, in accordance with some embodiments.

[0044] Figure 2 is a system diagram illustrating a set of example interfaces for an MPEG-I node hierarchy that supports scene interactivity elements, in accordance with some embodiments.

[0045] Figure 3 is a system diagram illustrating a set of example interfaces for an MPEG haptics architecture, in accordance with some embodiments.

[0046] Figure 4 is a system diagram illustrating example layered data structures for two codec formats, in accordance with some embodiments.

[0047] Figure 5 is a schematic illustration showing an example NAL unit structure in a haptic bitstream, in accordance with some embodiments.

[0048] Figure 6 is a schematic illustration showing example NAL unit payload types, in accordance with some embodiments.

[0049] Figure 7 is a code listing illustrating an example EditListBox class structure, in accordance with some embodiments.

[0050] Figure 8 is a schematic illustration showing an example MPD layered data model, in accordance with some embodiments.

[0051] Figure 9 is a system diagram illustrating an example DASH configuration for grouping adaptation sets, in accordance with some embodiments.

[0052] Figure 10 is a code listing illustrating an example XML schema according to some embodiments. Figure 9 Example data types for various elements and attributes of the XML schema are shown.

[0053] Figure 11 is a code listing illustrating an example XML schema according to some embodiments.

[0054] Figure 12 is a flowchart illustrating an example process for encoding haptic data according to some embodiments.

[0055] Figure 13 is a flowchart illustrating an example process for decoding haptic data according to some embodiments.

[0056] The entities, connections, arrangements, etc. depicted in the figures and described in connection with the figures are presented by way of example only, and not by way of limitation. Likewise, any and all statements or other indications of a specificity in regard to a particular figure, a particular element or entity in a particular figure “being” or “having” something, and any and all similar statements or other indications, when read in isolation and out of context, can be read as absolute and thus limiting, but can only be read correctly when read in context and in light of the constructive introductory phrases such as “in at least one embodiment,...” that precede them. For brevity and clarity of presentation, this implicit introductory phrase is not repeated at every occurrence of such statements or other indications in the detailed description. DETAILED DESCRIPTION

[0057] Figure 1A is a schematic diagram illustrating an example communications system 100 in which one or more disclosed embodiments can be implemented. The communications system 100 can be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communications system 100 can enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communications systems 100 can employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique-word OFDM (UW-OFDM), resource block-filtered OFDM, filter bank multicarrier (FBMC), and the like.

[0058] As Figure 1AAs shown, the communication system 100 can include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, a RAN 104, a CN 106, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, though it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d can be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d, any of which can be referred to as a “station” and / or a “STA”, can be configured to transmit and / or receive wireless signals, and can include a user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smartphone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (IoT) device, a watch or other wearable, a head-mounted display (HMD), a vehicle, a drone, a medical device and applications (e.g., remote surgery), an industrial device and applications (e.g., a robot and / or other wireless devices operating in an industrial and / or an automated processing chain context), a consumer electronics, a device operating on a commercial and / or industrial wireless network, and the like. Any of the WTRUs 102a, 102b, 102c, and 102d can be interchangeably referred to as a UE.

[0059] The communication system 100 can also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b can be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN 106, the Internet 110, and / or the other networks 112. By way of example, the base stations 114a, 114b can be a base transceiver station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a gNB, a NR NodeB, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each depicted as a single element, it will be appreciated that the base stations 114a, 114b can include any number of interconnected base stations and / or network elements.

[0060] The base stations 114a can be part of the RAN 104, which can also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base stations 114a and / or the base stations 114b can be configured to transmit and / or receive wireless signals on one or more carrier frequencies, which can be referred to as a cell (not shown). These frequencies can be in the licensed spectrum, the unlicensed spectrum, or a combination of the licensed and unlicensed spectrums. A cell can provide wireless service to a particular geographic area that can be relatively fixed, or can vary as users in the cell move throughout the network. The cell can further be divided into cell sectors. For example, a cell associated with a base station 114a can be divided into three sectors. Thus, in one embodiment, the base station 114a can include three transceivers, one for each sector of the cell. In an embodiment, the base station 114a can employ multiple-input multiple-output (MIMO) techniques, and can utilize multiple transceivers for each sector of the cell. For example, beamforming can be used to transmit and / or receive signals in desired spatial directions.

[0061] The base stations 114a, 114b can communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over the air interface 116, which can be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 can employ one or more wireless communication technologies, such as radio access technologies (RATs).

[0062] More specifically, as indicated above, the communication system 100 can be a multiple access system and can employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, the base station 114a in the RAN 104 and the WTRUs 102a, 102b, 102c can implement a radio technology, such as the Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which can establish the air interface 116 using wideband CDMA (WCDMA). WCDMA can include communication protocols, such as High-Speed Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA can include High-Speed Downlink (DL) Packet Access (HSDPA) and / or High-Speed UL Packet Access (HSUPA).

[0063] In the embodiment, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which can establish air interface 116 using Long Term Evolution (LTE) and / or Advanced LTE (LTE-A) and / or Advanced LTE Pro (LTE-A Pro).

[0064] In the embodiment, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as NR radio access, which can establish air interface 116 using new radio (NR).

[0065] In the embodiments, base station 114a and WTRUs 102a, 102b, and 102c can implement various radio access technologies. For example, base station 114a and WTRUs 102a, 102b, and 102c can, for example, use the dual connectivity (DC) principle to implement both LTE and NR radio access. Therefore, the air interface utilized by WTRUs 102a, 102b, and 102c can be characterized by various types of radio access technologies and / or transmissions sent to / from various types of base stations (e.g., eNBs and gNBs).

[0066] In other embodiments, base station 114a and WTRUs 102a, 102b, 102c can implement radio technologies such as IEEE 802.11 (i.e., Wi-Fi), IEEE 802.16 (i.e., Global Microwave Access Interoperability (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Provisional Standard 2000 (IS-2000), Provisional Standard 95 (IS-95), Provisional Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced GSM Evolution Data Rate (EDGE), GSM EDGE (GERAN), and so on.

[0067] Figure 1AThe base station 114b in the embodiment can be, for example, a wireless router, Home Node B, Home eNode B, or access point, and can utilize any suitable RAT for facilitating wireless connectivity access to the Internet, such as IEEE 802.11. In one embodiment, the base station 114b and the WTRUs 102c, 102d can implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In an embodiment, the base station 114b and the WTRUs 102c, 102d can implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d can utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish a picocell or femtocell. As shown, the base station 114b can have a direct connection to the Internet 110. Thus, the base station 114b can not be required to access the Internet 110 via the CN 106. Figure 1A

[0068] The RAN 104 can be in communication with the CN 106, which can be any type of network configured to provide voice, data, applications, and / or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data can have varying quality of service (QoS) requirements, such as differing throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, and the like. The CN 106 can provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, etc., and / or perform high-level security functions, such as user authentication. Although not shown in Figure 1A Although not shown in FIG. 1A, it will be appreciated that the RAN 104 and / or the CN 106 can be in direct or indirect communication with other RANs that employ the same RAT as the RAN 104 or a different RAT. For example, in addition to being connected to the RAN 104, which can be utilizing a NR radio technology, the CN 106 can also be in communication with another RAN (not shown) employing a GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0069] ​CN 106 can also serve as a gateway for WTRUs 102a, 102b, 102c, and 102d to access PSTN 108, the Internet 110, and / or other networks 112. PSTN 108 may include a circuit-switched telephone network providing Common Old-Style Telephone Service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices using common communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and / or Internet Protocol (IP) from the TCP / IP Internet Protocol Suite. Network 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, network 112 may include another CN connected to one or more RANs, which may use the same RAT as RAN 104 or a different RAT.

[0070] Some or all of the WTRUs 102a, 102b, 102c, and 102d in communication system 100 may include multi-mode capabilities (e.g., WTRUs 102a, 102b, 102c, and 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). For example, Figure 1A The WTRU 102c shown can be configured to communicate with base station 114a, which may employ cellular-based radio technology, and with base station 114b, which may employ IEEE 802 radio technology.

[0071] Figure 1B The following diagram illustrates the system of example WTRU 102. Figure 1B As shown, among other things, WTRU 102 may include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power supply 134, a Global Positioning System (GPS) chipset 136, and / or other peripheral devices 138. It will be appreciated that WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with the embodiments.

[0072] The processor 118 can be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 can perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 can be coupled to the transceiver 120, which can be coupled to the transmit / receive element 122. While Figure 1B The processor 118 and the transceiver 120 are depicted as separate components, it will be appreciated that the processor 118 and the transceiver 120 can be integrated together in an electronic package or chip.

[0073] The transmit / receive element 122 can be configured to transmit signals to, or receive signals from, a base station (e.g., the base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 can be an antenna configured to transmit and / or receive RF signals. In an embodiment, the transmit / receive element 122 can be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit / receive element 122 can be configured to transmit and / or receive both RF and light signals. It will be appreciated that the transmit / receive element 122 can be configured to transmit and / or receive any combination of wireless signals.

[0074] Although the transmit / receive element 122 is depicted in the WTRU 102 Figure 1B In one embodiment, the WTRU 102 can include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0075] The transceiver 120 can be configured to modulate information to be transmitted by the transmit / receive element 122 and to demodulate information received by the transmit / receive element 122. As indicated above, the WTRU 102 can include a plurality of transceivers 120 for enabling the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11, for example.

[0076] The processor 118 of the WTRU 102 can be coupled to, and can receive user input data from, the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit). The processor 118 can also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 can access information from, and store data in, any type of suitable memory, such as the non-removable memory 130 and / or the removable memory 132. The non-removable memory 130 can include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 can include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 can access information from, and store data in, memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[0077] The processor 118 can receive power from the power source 134, and can be configured to distribute and / or control the power to the other components in the WTRU 102. The power source 134 can be any suitable device for powering the WTRU 102. For example, the power source 134 can include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0078] The processor 118 can also be coupled to the GPS chipset 136, which can be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or

[0079] The processor 118 can further be coupled to other peripherals 138 that can include one or more software and / or hardware modules that provide additional features, functionality and / or wired or wireless connectivity. For example, the peripherals 138 can include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photographs and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands -free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, and the like. The peripherals 138 can include one or more sensors, the sensors can be one or more of a gyroscope, an accelerometer, a hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0080] The WTRU 102 can include a full duplex radio for which transmission and reception of some or all signals associated with the WTRU 102 can be concurrent and / or simultaneous. The full duplex radio can include an interference management unit to reduce and / or eliminate self-interference and / or cross- interference due to concurrent or simultaneous transmission and reception. In an embodiment, the WTRU 102 can include a half duplex radio for which transmission and reception of some or all signals associated with the WTRU 102 are time divided.

[0081] Although the WTRU is described in Figures 1A-1B as a wireless terminal, it is contemplated that in certain representative embodiments that such a terminal can use (e.g., temporarily or permanently) wired communication interfaces with the communication network.

[0082] In representative embodiments, the other network 112 can be a WLAN.

[0083] In view of Figures 1A-1B the corresponding description, one or more or all of the functions described herein can be performed by one or more emulation devices (not shown). The emulation device can be one or more devices configured to emulate one or more or all of the functions described herein. For example, the emulation device can be used to test other devices and / or to simulate network and / or WTRU functionality.

[0084] Simulation devices can be designed to perform one or more tests on other devices in laboratory and / or carrier network environments. For example, one or more simulation devices can perform one or more functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices within the communication network. One or more simulation devices can perform one or more functions while being temporarily implemented / deployed as part of a wired or wireless communication network. Simulation devices can be directly coupled to another device for testing purposes and / or can perform tests using over-the-air wireless communication.

[0085] One or more simulation devices can perform one or more functions without being implemented / deployed as part of a wired and / or wireless communication network. For example, simulation devices can be used to test scenarios in laboratory and / or undeployed (e.g., test) wired and / or wireless communication networks to enable testing of one or more components. One or more simulation devices can be test rigs. Simulation devices can transmit and / or receive data using direct RF coupling and / or wireless communication via RF circuitry (e.g., which may include one or more antennas).

[0086] The embodiments described herein are not limited to implementation on WTRU. Such embodiments can be implemented using other systems, such as Figure 1C The system.

[0087] Figure 1C This is a system diagram illustrating a set of example interfaces for a system according to some embodiments. Interfaces such as... Figure 1C Systems such as these implement extended reality display devices and their control electronics. System 140 can be embodied as a device including the various components described below and configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 140 can be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 140 are distributed across multiple ICs and / or discrete components. In various embodiments, system 140 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 140 is configured to implement one or more aspects described in this document.

[0088] The system 140 includes at least one processor 142 configured to execute instructions loaded therein for implementing, e.g., the various aspects described in this document. Processor 142 can include embedded memory, input output interface, and various other circuitry known from the art. The system 140 includes at least one memory 144 (e.g., a volatile memory device and / or a non-volatile memory device). The system 140 can include a storage device 148, which can include non-volatile memory and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drive, and / or optical disk drive. By way of non-limiting example only, the storage device 148 can include an internal storage device, an attached storage device (including detachable and non-detachable storage devices), and / or a network accessible storage device.

[0089] The system 140 includes an encoder / decoder module 146 configured, e.g., to process data to provide encoded video or decoded video, and the encoder / decoder module 146 can include its own processor and memory. The encoder / decoder module 146 is representative of the module(s) that can be included in a device to perform the encoding and / or decoding functions. As is known, a device can include one or both of the encoding and decoding modules. Furthermore, the encoder / decoder module 146 can be implemented as a standalone element, or can be incorporated as part of the processor 142 as a combination of hardware and software as known to those skilled in the art.

[0090] Program code to be loaded onto processor 142 or encoder / decoder 146 to perform the various aspects described in this document can be stored in the storage device 148 and then loaded onto the memory 144 for execution by the processor 142. In accordance with various embodiments, one or more of the processor 142, the memory 144, the storage device 148, and the encoder / decoder module 146 can store one or more of various items during the performance of the processes described in this document. Such stored items can include, without limitation, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0091] In some embodiments, the processor 142 and / or memory internal to the encoder / decoder module 146 are used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device can be the processor 142 or the encoder / decoder module 142) is used for one or more of these functions. The external memory can be the memory 144 and / or storage device 148, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store, for example, an operating system of the television. In at least one embodiment, fast external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by the Joint Video Expert Team, JVET).

[0092] As indicated by block 162, input can be provided to elements of the system 140 through various input devices. Such input devices include, but are not limited to, (i) a radio frequency (RF) portion that receives RF signals transmitted over the air, for example, by a broadcaster, (ii) a component (COMP) input terminal (or set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 1C Other examples, not shown in FIG. 1, include composite video.

[0093] In various embodiments, the input devices of block 162 have associated respective input processing elements known in the art. For example, the RF portion can be associated with elements suitable for: (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a frequency band of frequencies), (ii) downconverting the selected signal, (iii) band-limiting again to a narrower band of frequencies to select a signal frequency band which may, e.g., be referred to as a channel in certain embodiments, (iv) demodulating the downconverted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF portion of various embodiments includes one or more elements to perform these functions, e.g., frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion can include a tuner to perform various ones of these functions, including, e.g., downconverting the received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing elements receive an RF signal transmitted through a wired (e.g., cable) medium and perform frequency selection by filtering, downconverting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements performing similar or different functions. Adding elements can include inserting elements in between existing elements, such as, e.g., inserting amplifiers and analog-to-digital converters. In various embodiments, the RF portion includes an antenna.

[0094] Additionally, the USB and / or HDMI terminals can include respective interface processors for connecting the system 140 to other electronic devices across USB and / or HDMI connections. It will be appreciated that various aspects of input processing, e.g., Reed-Solomon error correction, can be implemented in, e.g., separate input processing ICs or in the processor 142 as desired. Similarly, aspects of USB or HDMI interface processing can be implemented within separate interface ICs or within the processor 142 as desired. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, e.g., the processor 142 and the encoder / decoder 146, which operate in conjunction with memory and storage elements to process the data streams as desired for presentation on output devices.

[0095] The various elements of the system 140 can be provided within an integrated housing, within which the various elements can be interconnected and transfer data therebetween using a suitable connection arrangement 164, e.g., an internal bus known in the art, including an Inter-IC (I2C) bus, wiring, and printed circuit boards.

[0096] The system 140 includes a communication interface 150 that enables communication with other devices via a communication channel 152. The communication interface 150 can include, but is not limited to, a transceiver configured to transmit and receive data through the communication channel 152. The communication interface 150 can include, but is not limited to, a modem or network card, and the communication channel 152 can be implemented in, for example, a wired and / or wireless medium.

[0097] In various embodiments, data is streamed or otherwise provided to the system 140 using a Wi-Fi network such as, for example, IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signals of these embodiments are received through the communication channel 152 and the communication interface 150 adapted for Wi-Fi communication. The communication channel 152 of these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments provide streaming data to the system 140 using a set-top box that communicates data through an HDMI connection of the input block 162. Still other embodiments provide streaming data to the system 140 using an RF connection of the input block 162. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0098] The system 140 can provide output signals to various output devices, including a display 166, speakers 168, and other peripheral devices 170. The display 166 of various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 166 can be used in a television, a tablet, a laptop, a cellular phone (mobile phone), or other device. The display 166 can also be integrated with other components (e.g., as in a smartphone), or be standalone (e.g., an external monitor for a laptop). In various examples of embodiments, the other peripheral devices 170 include one or more of a standalone digital video recorder (or digital versatile recorder) (DVR, for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more of the peripheral devices 170 that provide functionality based on the output of the system 140. For example, a disc player performs the functionality of playing the output of the system 140.

[0099] In various embodiments, signaling using a communication protocol such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols capable of enabling device-to-device control with or without user intervention, communicates control signals between system 140 and display 166, speakers 168, or other peripheral devices 160. The output devices can be communicatively coupled to system 140 via the respective interfaces 154, 156, and 158 through dedicated connections. Alternatively, the output devices can be connected to system 140 using the communication channel 152 via the communication interface 150. The display 166 and speakers 168 can be integrated in a single unit with other components of system 140 in an electronic device, such as, for example, a television. In various embodiments, the display interface 154 includes a display driver, such as, for example, a timing controller (T Con) chip.

[0100] For example, if the RF portion of input 162 is part of a separate set-top box, the display 166 and speakers 168 can be separate from one or more other components. In various embodiments where the display 166 and speakers 168 are external components, the output signals can be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0101] System 140 can include one or more sensor devices 160. Examples of sensor devices that can be used include one or more GPS sensors, gyroscope sensors, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. Such sensors can be used to determine information such as user location and orientation. In the case where system 140 is used as a control module for an extended reality display, such as a control module for a head-mounted display device, the user’s location and orientation can be used to determine how to render image data so that the user perceives the correct portion of a virtual object or virtual scene from the correct viewpoint. In the case of a head-mounted display device, the location and orientation of the device itself can be used to determine the user’s location and orientation for purposes of rendering virtual content. In the case of other display devices, such as telephones, tablets, computer monitors, or televisions, other inputs can be used to determine the user’s location and orientation for purposes of rendering content. For example, the user can use a touch screen, keypad, or keyboard, trackball, joystick, or other input to select and / or adjust a desired viewpoint and / or viewing direction. In the case where a display device has sensors such as accelerometers and / or gyroscopes, the viewpoint and orientation can be selected and / or adjusted based on motion of the display device for purposes of rendering content.

[0102] The embodiments can be implemented by computer software implemented by the processor 142 or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. The memory 144 can be of any type appropriate for the technology environment and can be implemented using any suitable data storage technology, as non-limiting examples, such as optical, magnetic, semiconductor-based, fixed, removable or solid state storage. The processor 142 can be of any type appropriate for the technology environment, and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as non-limiting examples.

[0103] Runtime Interactivity Figure 2 is a system diagram illustrating a set of example interfaces of the MPEG-I node hierarchy 200 that support scene interactivity elements according to some embodiments. According to the present principles, in addition to the node tree as described with respect to Figure 3

[0104] In some embodiments, the behaviors are time-evolving. In such embodiments, the behaviors can be updated through the already existing scene description update mechanisms.

[0105] In example embodiments, a behavior is characterized by one or more of the following properties: • one or more triggers defining conditions to be fulfilled for activation.

[0106] • trigger control parameters defining logical operations between defined triggers.

[0107] • actions to be implemented in response to activation of the triggers.

[0108] • action control parameters defining the execution order of defined actions.

[0109] • a priority number enabling selection of the highest priority behavior in case several behaviors occur simultaneously on the same virtual object.

[0110] • an optional interrupt action to specify how to terminate the behavior when the behavior is no longer defined in newly received scene updates. For example, if the relevant object has been removed, or if the behavior is no longer relevant to the current media (e.g., audio or video) sequence, the behavior is no longer defined. ​

[0111] By adding these behaviors, time-dependent user interactivity in immersive content for XR experiences can be defined.

[0112] When a second scene description is received, some behaviors of the first scene description can be "in progress", i.e. they are triggered and their actions are running. The second scene description can be provided as update metadata, which is metadata describing the difference between the first scene description and the second description. The second scene description includes a tree of nodes describing objects that can be common or different from the objects of the first scene description. Objects of the tree of nodes of the first scene description can no longer be present in the second description. If the objects related to the running actions of the in-progress behaviors are missing in the second scene description, then these in-progress behaviors are no longer applicable. Likewise, if an in-progress behavior is not defined in the second description, then the in-progress behavior is no longer applicable. The interrupt action field describes how to properly interrupt the running actions of the in-progress behaviors.

[0113] Encoding haptic data Figure 3 is a system diagram illustrating a set of example interfaces of the MPEG haptics architecture, according to some embodiments. Figure 3 An example haptics codec architecture 300 is shown. A new standard is currently being developed by the Moving Picture Experts Group (MPEG), WG07N00624, Text of ISO / IEC DIS 23090-31 MPEG Haptics Coding Part 1 , MPEG 142 (April 2023) ("MPEG-H 3D Audio"). ISO / IEC 23090-31 MPEG-H 3D Audio"). Figure 3 The MPEG haptics codec architecture 300 is illustrated. See ISO / IEC 23090-31 .

[0114] In this architecture 300, the coded representation of haptic data can be in one of two formats: an interchange format (.hjif) 302 or a distribution format (.hmpg) 304. The interchange format is a JSON-based human-readable description of the haptic data, while the distribution format is a compressed binary representation of the data. These two formats have complementary purposes and can be executed to perform a lossless or lossy one-to-one conversion between them. Examples of this conversion are shown in Figure 3 and around the center of the "interchange format" term.

[0115] The compressed binary bitstream in the distribution format is structured into a sequence of Network Abstraction Layer (NAL) units, called "packets", to facilitate sealing over any network protocol or file format. This operation is illustrated with a binary compression block 306 and a binary decompression block 308, and intermediate blocks.

[0116] The haptic decoder 310 takes a binary ".hmpg" file or ".hjif" as input and outputs a ".hjif" file. The haptic data contained in the resulting ".hjif" file can be rendered directly on a haptic device or using an intermediate synthesizer 312 that generates pulse code modulated (PCM) data, as shown on the right side of Figure 3

[0117] Figure 4 is a system diagram illustrating example hierarchical data structures for two codec formats, according to some embodiments. Figure 4 An example haptic data hierarchy 400 is shown. The data structures for the two haptic codec formats follow the hierarchical organization illustrated in Figure 4

[0118] The highest level of the structure describes the overall haptic experience defined in a file or stream. This highest level contains some high-level metadata information 402 and provides a list of avatars 404 and / or body representations to which the desired location of haptic stimulation on the body can be referenced. The haptic data itself is described by a list of Perception

[0119] In addition to the sensation-specific metadata 408, a sensation can include a list of channels (or tracks 410) in which the data is broken down into frequency bands. Each frequency band 412 defines a portion of the signal in a given frequency range. The frequency bands 412 are described with a list of haptic effects 414, each including a list of keyframes 416. The haptic signal in a channel can be reconstructed by combining the data in the different frequency bands (e.g., by adding a high frequency band and a low frequency band).

[0120] Figure 5 ​​​is a schematic illustration showing an example NAL unit structure in a haptic bitstream according to some embodiments. The NAL unit (NALu) structure 500 includes a header and a payload. The NALu header 502 is 32 bits, including 4 bits for a nal_unit_type field, 2 bits for a level field, 10 bits reserved for future use, and 16 bits for a payload byte length value. The nal_unit_type field can indicate a metadata experience type (including nb perception, stage 1, 2a or 2b or avatar); a metadata perception type (including nb track, type, library or device); a metadata track type (including nb frequency band) or a data frequency band type (including header frequency band). The level field is a frequency band level with 0 as a baseline. The metadata NALu payload 504 is n bits of metadata, where n is a variable. The data frequency band NALu payload 506 is a variable length field, where n*8 bits of header and n groups of subfields each n bits wide, FX1 to FXn.

[0121] Figure 6 is a schematic illustration showing example NAL unit payload types according to some embodiments. Figure 6 An example structure 600 is shown for how the NALu header 602 and NALu payload 604 structure is implemented for several NAL unit types. Figure 6 The left side of shows a 16 bit NALu header 602, where 4 bits indicate the NAL type, 2 bits indicate the level, and 10 bits are reserved for future use. The NAL type of b0000 is shown as corresponding to metadata haptic experience data 606. The NAL type of b0001 is shown as corresponding to metadata haptic perception data 608. The NAL type of b0010 is shown as corresponding to metadata haptic track data 610. The NAL type of b0011 is shown as corresponding to metadata haptic frequency band data 612. The NAL type of b0100 is shown as corresponding to effect database 614. The NAL type of b0101 is shown as corresponding to data frequency band data 616. The NAL type of b1100 is shown as corresponding to a cyclic redundancy code (CRC) 618. The NAL type of b1101 is shown as corresponding to a byte buffer 620.

[0122] ISO base media file format Figure 7 is a code listing illustrating an example EditListBox class structure according to some embodiments. Figure 7 An example code listing 700 is shown. Within the ISO / IEC 14496 (MPEG-4) standard, there are several parts that define file formats for storage of time-based media. According to ISO / IEC 14496-12, Coding of audio-visual objects - Part 12: ISO base media file formatISO / IEC 14496-12 , 2020 (“ ISO / IEC 14496-12 ”), these parts are all based on and derived from the ISO Base Media File Format (ISOBMFF), which is a structured, media-independent definition. The ISOBMFF contains structured and media data information for timed presentation of media data such as audio, video, etc. Non-timed data is also supported, such as metadata at different levels within the file structure. The logical structure of a file is a movie, which in turn contains a set of time-parallel tracks. The temporal structure of a file is such that a track contains a sequence of samples in time, and these sequences are mapped into the timeline of the overall movie. The ISOBMFF is based on the concept of box-structured files. A box-structured file has a series of boxes (sometimes called atoms) that have a size and a type. The type is a 32-bit value, and is usually chosen to be four printable characters, also called a four-character code (4CC). Non-timed data can be contained in a metadata box at the file level, or attached to one of the timed data streams (called tracks) within the movie box or movie.

[0123] Between the top-level boxes in an ISOBMFF container is the MovieBox (“moov”), which contains metadata for the continuous media streams present in the file. This metadata is signaled within the hierarchy of boxes within the movie box, for example within a TrackBox (“trak”). A track represents a continuous media stream that is presented in the file. The media stream itself is a sequence of samples, such as audio or video access units of an elementary media stream, and is enclosed within a MediaDataBox (“mdat”) that is present at the top level of the container. The metadata for each track includes a list of sample description entries, each of which provides the coding or encapsulation format used in the track and initialization data for processing that format. Each sample is associated with one of the sample description entries for the track. Figure 7 A tool is provided for defining an explicit timeline mapping for each track. This is called an edit list, and is signaled using an EditListBox with the syntax shown in Figure 7 Each entry defines a portion of the track timeline: either by mapping a portion of the timeline, or by indicating a “null” time (a portion of the presentation timeline that is not mapped to media, a “null” edit).

[0124] Figure 7 An example class structure for the EditListBox extension of the FullBox class is shown. For the example in Figure 7 , the sequence of samples numbered 1 to the entry_count value form the media stream. Text of ISO / IEC CD 23090-32 Carriage of Haptics DataSupport for "Version 0" and "Version 1" is shown. Version 0 uses 32-bit integers for the duration of a sample (variable edit_duration) and time within a media stream (variable media_time), while Version 1 uses 64-bit integer fields. Each sample also has media_rate_integer and media_rate_fraction values that correspond to the sampling rate of the sample.

[0125] Carrying of haptic data MPEG is currently working on developing a new standard WG03N00686, ISO / IEC 23090-32 ISO / IEC , MPEG 142, April 2023 ( ISO / IEC CD 23090-32 ), which defines how the haptic bitstream generated by [Touch] on Carriage of Haptics Data in ISOBMFF 23090-31 a codec can be sealed in an ISOBMFF media container. ISO / IEC 23090-31 The current version of ISO / IEC 23090-31 describes how to carry haptic data in a single track in a file.

[0126] The '638 application describes a more flexible and extensible design for carrying haptic data using a multi-track approach. In the multi-track mode, a haptic experience is carried in multiple ISOBMFF tracks, with one track being the main track of the experience and carrying general information that applies to the overall experience. The main track can be associated with one or more haptic tracks through track references in the ISOBMFF file, which carry haptic band data units for one or more haptic channels. The samples of that track can carry data for only one band of a perceptual channel or can carry data for all bands of a perceptual channel. If all bands are carried in the samples of the haptic track, each sample can be composed of sub-samples, with each sub-sample containing data for one of the bands.

[0127] While the bitstream format developed in ISO / IEC 23090-31 enables a haptic experience to be described in a succinct representation that can be easily consumed by a haptic decoder, there is currently no well-defined or standardized approach for streaming such coded haptic content. In m 61136, Figure 8, MPEG 140, October 2022, presented a method to store ISO / IEC 23090-31 coded haptic bitstreams in an ISOBMFF container by demultiplexing data belonging to different haptic channels into separate tracks in the file. While this design allows for scalable access to different haptic stream components in a local playback scenario, such as when all data is stored and accessed locally, such a structure cannot be directly used to enable remote access over a network and adaptive streaming of haptic content stored on a remote server.

[0128] This application presents methods for supporting adaptive streaming of haptic bitstreams generated by Element and attribute names ISO / IEC 23090-31 and encapsulated in an ISOBMFF container as multiple tracks. For some embodiments, this application describes methods and systems to enable flexible and scalable streaming of haptic media, which can be coded using the ISO / IEC 23090-31 codec developed by ISO / IEC SC29 / WG03 (MPEG Systems).

[0129] For some embodiments, the concepts presented in this application can be applied to immersive media coding, encoding, streaming of haptic media content, and decoding of haptic media data on devices or any services that provide immersive media experiences.

[0130] Dynamic adaptive streaming over HTTP (DASH) MPEG Dynamic Adaptive Streaming over HTTP (MPEG-DASH) is a versatile delivery format that provides the best possible video experience for end users by dynamically adapting to changing network conditions. Dynamic HTTP streaming requires various bitrate alternatives of multimedia content to be available at the server. In addition, multimedia content can be composed of several media components (e.g., audio, video, text), each of which can have different characteristics. In MPEG-DASH, these characteristics are described by a Media Presentation Description (MPD).

[0131] Use is a schematic illustration showing an example MPD hierarchical data model according to some embodiments. The example hierarchical data model 800 includes a series of hierarchical boxes. The MPD 802 describes a Period sequence 804, where a consistent set of encoded versions of media content components do not change during one period. Each period has a start time and duration, and is composed of one or more Adaptation Sets.

[0132] An Adaptation Set 806 represents a set of encoded versions of one or several media content components that share the same properties, such as language, media type, picture aspect ratio, role, accessibility, and rating properties. For example, one AdaptationSet can contain different bitrates of the video component of the same multimedia content. Another AdaptationSet can contain different bitrates of the audio component of the same multimedia content (e.g., lower quality stereo and higher quality surround sound). Each AdaptationSet typically includes multiple Representations.

[0133] A Representation 808 describes a deliverable encoded version of one or several media components, differing from other representations in terms of bit rate, resolution, number of channels, or other characteristics. Each Representation is composed of one or more Segments. Attributes of the Representation element such as @id, @bandwidth, @qualityRanking, and @dependencyld are used to specify the properties of the associated Representation. A Representation can also include sub-representations, which are parts of a Representation used to describe and extract partial information from the Representation. Sub-representations can provide the ability to access lower quality versions of the Representation in which they are contained.

[0134] A Segment 810 is the largest unit of data that can be retrieved with a single HTTP request. Each Segment has a URL, an addressable location on a server, which can be downloaded using HTTP GET or HTTP GET with byte ranges.

[0135] To use this data model, a DASH client parses the MPD XML document and, based on the information provided in each AdaptationSet element, selects a set of AdaptationSet that fits its environment. Within each AdaptationSet, the client selects a Representation, typically based on the value of the @bandwidth attribute, but also taking into account the client decoding and rendering capabilities. The client downloads the initialization segment of the selected Representation and then accesses the content by requesting whole segments or byte ranges of segments. Once the presentation has started, the client continues to consume the media content by continuously requesting media segments or parts of media segments and plays the content according to the media presentation timeline. The client can switch Representation, taking into account updated information from its environment. The client should continuously play the content across periods. Once the client is consuming media contained in a segment when it reaches the end of the media announced in the Representation, then the media presentation is terminated, a new period starts, or the MPD can be reacquired.

[0136] Descriptors in DASH MPEG-DASH introduces the concept of descriptors to provide application-specific information about the media content. Descriptor elements are all structured in the same way, i.e. they contain an @schemeIdUri attribute (which provides a URI to identify the scheme) and an optional attribute @value and an optional attribute @id. The semantics of the element are specific to the scheme employed. The URI identifying the scheme can be a URN or a URL. The MPD does not provide any specific information on how to use these elements. It depends on the application that employs the DASH format to instantiate the descriptor elements with the appropriate scheme information. A DASH application using one of these elements must first define the scheme identifier in the form of a URI and then must define the value space of the element when this scheme identifier is used. If structured data is required, then any extension elements or attributes can be defined in a separate namespace. Descriptors can appear at multiple levels within the MPD: • The element is present at the MPD level means that the element is a child of the MPD element.

[0137] • The element is present at the AdaptationSet level means that the element is a child of the AdaptationSet element.

[0138] • The element is present at the Representation level means that the element is a child of the Representation element.

[0139] Preselection In MPEG-DASH, a bundle is a set of media components that can be consumed collectively by a single decoder instance. Each bundle includes a main media component that contains decoder-specific information and directs the decoder. PreSelection defines a subset of media components in a bundle that is intended to be consumed collectively.

[0140] An AdaptationSet that contains a main media component is referred to as a main AdaptationSet. The main media component is always included in any PreSelection associated with a bundle. In addition, each bundle can include one or more partial AdaptationSets. A partial AdaptationSet can only be processed in combination with a main AdaptationSet.

[0141] A PreSelection can be defined by a PreSelection element as defined in Table 1. The selection of a PreSelection is based on the attributes and elements contained in the PreSelection element. For Table 1, attributes that are designated as mandatory are listed with a Use value of "M", while attributes that are designated as optional are listed with a Use value of "O". An "OD" entry means that the attribute is optional with a default value listed. For elements, the Use value lists the minimum occurrence to the maximum occurrence. An entry of "N" means that the occurrence is not limited. Elements are shown in bold first, while attributes are shown in non-bold and start with an "@" symbol. Description PreSelection: @id OD default value = 1 Specifies the id of the PreSelection. This shall be unique within a period. @preselectionComponents Specifies the ids of the contained adaptation sets or content components that belong to this PreSelection as a space-separated list in processing order, where the first id is the id of the main media component. @lang M Declares the language code used for the PreSelection according to the syntax and semantics in IETF RFC 5646. Accessibility O Specifies information about the accessibility scheme. Role 0 … N Specifies information about the role annotation scheme. Rating 0 … N Specifies information about the rating scheme. Viewpoint 0 … N Specifies information about the viewpoint annotation scheme. CommonAttributesElements 0 … N Specifies common attributes and elements (attributes and elements from the base type RepresentationBaseType). Figure 9 - Figure 9

[0142] Adaptation sets for haptic media Elements and attributes is a system diagram illustrating an example DASH configuration for grouping adaptation sets, according to some embodiments. Use An example of a DASH configuration 900 for grouping adaptation sets 904, 906, 908, 910 within an MPEG-DASH MPD file that belong to the same haptic experience 902 is illustrated.

[0143] To signal the presence of multi-track haptic media in a DASH Media Presentation Descriptor (MPD), each track of the media (including the main track) can be represented by an AdaptationSet element in the MPD. The AdaptationSet 904 of the main (haptic experience) track is referred to as the haptic experience AdaptationSet, and the AdaptationSets 906, 908, 910 of the associated haptic tracks are referred to as the haptic AdaptationSets.

[0144] 4CC is a 4-character code used to identify the type of sample entry of a track in ISOBMFF, which in turn is based on the type of codec (e.g., "hev1" is a 4CC for a HEVC codec with some configuration). A track is represented by an Adaptation Set in a DASH MPD. Typically, the @codecs attribute is set to the corresponding 4CC of the associated track. For some embodiments, the same 4CC can be used for both the haptic experience AS and the haptic data AS, as they are using the same haptic codec.

[0145] For some embodiments, a perception is part of the same haptic experience. A haptic experience incorporates multiple perceptions, and each perception has multiple channels, and each channel has multiple frequency bands. A haptic data adaptation set represents a track that carries frequency band data for one or more channels of a certain perception in a haptic experience.

[0146] For each haptic adaptation set (or Representation of these adaptation sets), the haptic experience adaptation set sets the @codecs attribute to "mih1", and if the @codecs attribute is not presented in the AdaptationSet element, the @codecs attribute is set to "mihb". The @mimeType attribute of all adaptation sets of a haptic experience is set to "haptic / mp4", which is the MIME type registered for haptic media.

[0147] A haptic experience adaptation set contains a single initialization segment at the adaptation set level. The initialization segment shall contain all MPEG-l Haptic Stream (MIHS) units (packets) needed to initialize the haptic decoder. The media segments of the Representation of a haptic adaptation set contain one or more track fragments of the corresponding haptic track at the file format level. By concatenating the initialization segment with the media segments from one or more haptic adaptation sets, the resulting file contains a bitstream that can be decoded by a haptic decoder.

[0148] The Representation of a haptic adaptation set preselected by a haptic experience shall set the @dependencyld attribute to the id of the Representation in the corresponding haptic experience adaptation set.

[0149] If a haptic adaptation set contains more than one Representation, the @bitstreamSwitching attribute is present in the AdaptationSet element of the haptic adaptation set and is set to "true" to indicate to the player that seamless switching between the Representations in the adaptation set is supported. In addition, the duration of the media segments in each Representation must be the same.

[0150] For example, if the segments are not the same, then if the player switches to a different representation, the player has to figure out where the player should continue playback in the new media segments retrieved from the other representation. The player also needs to calculate which segment the player needs from the last timestamp of the last segment (played by the player from the first representation). Thus, in this case, the switch will not be seamless. The segment index will not align in time. Therefore, according to some embodiments, the duration of the media segments in each Representation must be the same.

[0151] Haptic experience preselection Haptic experience preselection can be signaled in the MPD using a PreSelection element within the Period element or using a preselection descriptor at the adaptation set level. The haptic experience PreSelection element is signaled with the @preselectionComponents attribute as defined in ISO / IEC 23009-1, whose assigned value is a list of ids that includes the ids of the haptic experience adaptation sets followed by the id of the associated haptic adaptation set. The @codecs attribute for preselection is set to "mihl" indicating that the media represented by the preselection is coded haptic media.

[0152] For some embodiments, an adaptation set can include information identifying one or more segments of a corresponding haptic track.

[0153] Haptic experience descriptor Table 2 lists the elements and attributes of the haptic descriptor, which can be contained in a separate file. For Table 2, attributes designated as mandatory are listed with a Use value of "M", while attributes designated as optional are listed with a Use value of "O". For each element, the Use value lists the minimum number of occurrences to the maximum number of occurrences. An entry of "N" means that the number of occurrences is not limited. Elements are shown in bold first, while attributes are shown in non-bold and begin with a "@" symbol.

[0154] To signal different perceptions within a haptic experience, a HapticsExperience descriptor is created for the haptic experience adaptation set. In some embodiments, the HapticsExperience descriptor is an EssentialProperty descriptor with its @schemeIdUri attribute set to a unique Uniform Resource Identifier (URI) (e.g., "urn:mpeg:mpegI:haptics:2023:perception"). The @value attribute of the HapticsExperience descriptor should not be rendered. The HapticsExperience descriptor includes elements and attributes that describe the haptic experience and associated perceptions. The haptic descriptor includes at least one or more hashesPerception elements, each including an @id attribute set as a unique identifier for the perception in the haptic bitstream and an @type attribute used to signal its modality. Data type Description hapticsPerception haptics:HapticsPerceptionType An element whose attributes specify information of one of the haptic perceptions presented in the haptic experience. 0…N hapticsPerception@id xs:string An identifier indicating the perception. The value shall match the perception_id field signaled in the ISOBMFF container for the perception. M hapticsPerception@type xs:string An identifier indicating the perception the channel belongs to. O Figure 10 Figure 10 Table 2.

[0155] Figure 10 This is a list of code illustrating example XML schemas according to some embodiments. Elements and attributes A list of 1000 example codes showing various elements and attributes for XML schemas is provided. Use In the example shown, line 6 of the code list sets the element name to "hapticsPerception" and the data type to "haptics:HapticsPerceptionType". Line 8 of the code list sets the name, type, and usage of the @id attribute. Line 9 of the code list sets the name, type, and usage of the @type attribute.

[0156] In some implementations, the HapticsExperience descriptor is a SupplementalProperty descriptor with the @schemeIdUri attribute set to a unique URI (e.g., "urn:mpeg:mpegI:haptics:2023:perception").

[0157] Tactile channel descriptor Table 3 lists the elements and attributes of the haptic descriptor, which can be included in a separate file. For Table 3, attributes designated as mandatory are listed using the Use value "M", while attributes designated as optional are listed using the Use value "O". For elements, the Use value lists the minimum to maximum number of occurrences. An entry "N" means there is no limit to the number of occurrences. Elements are first shown in bold, while attributes are shown in non-bold and begin with the "@" symbol.

[0158] To identify the haptic channels presented in a haptics adaptation set, a haptics descriptor is used. For some embodiments, the haptics descriptor is an EssentialProperty descriptor with @schemeIdUri set to a unique URI (e.g., “urn:mpeg:mpegI:haptics:2023:channel”). At the adaptation set level, for each haptic channel presented in the Representation of a haptics adaptation set, the haptics descriptor is signaled. Again, the @value attribute of the haptics descriptor should not be presented. The haptics descriptor is associated with a haptics data adaptation set and provides information about the channel(s) and band(s) carried in that adaptation set. The HapticsExperience descriptor provides higher level metadata about the haptics experience itself. In this case, a list of perceptions available in the experience. In the haptics descriptor, the hapticChannel element refers to one of these perceptions. Data type Description ​ ​ hapticChannel 0 … N haptics:HapticsChannelType An element whose attributes specify information for one of the haptic channels presented in the Representation(s) of the adaptation set. hapticChannel@id M xs:string An identifier indicating the channel. The value matches the channel_id field signaled for the channel in the ISOBMFF container. hapticChannel@perceptionId M xs:string An identifier indicating the perception to which the channel belongs. hapticChannel@band_ids O xs:UIntVectorType Specifies the frequency bands related to the haptic data contained in the adaptation set for the haptic channel with the identifier equal to the value of the @id attribute by providing a space-separated list of frequency band ID values. If this attribute is not presented, then the adaptation set contains all the frequency bands associated with the channel with the identifier equal to the @id value. Table 3.

[0159] Figure 11 is a code listing illustrating an example XML schema according to some embodiments. Figure 11 An example code listing 1100 is shown. For this example, Figure 10 For the example shown, data types are defined in the XML schema for various elements and attributes. For this example, Figure 11 For the example shown in, line 6 of the code listing sets the element name to “hapticsChannel” and the data type to “haptics:HapticsChannelType”. Line 8 of the code listing sets the name, type, and use of the @id attribute. Line 9 of the code listing sets the name, type, and use of the @perceptionId attribute. Line 10 of the code listing sets the name, type, and use of the @band_id attribute.

[0160] Figure 12 is a flowchart illustrating an example process for encoding haptics data according to some embodiments. For some embodiments, the example process 1200 can include encoding 1202 information describing a haptics experience. For some embodiments of the example process 1200, the information describing 1204 the haptics experience includes one or more adaptation sets. For some embodiments of the example process 1200, each of the one or more adaptation sets includes 1206 one or more representations corresponding to a haptics media track.

[0161] Figure 13is a flowchart illustrating an example process for decoding haptic data, in accordance with some embodiments. For some embodiments, the example process 1300 can include decoding 1302 information describing a haptic experience. For some embodiments of the example process 1300, the information describing 1304 the haptic experience includes one or more adaptation sets. For some embodiments of the example process 1300, each of the one or more adaptation sets includes 1306 one or more representations corresponding to a haptic media track.

[0162] While the methods and systems according to some embodiments are generally discussed in the context of extended reality (XR), some embodiments can be applied to any XR context, such as, for example, virtual reality (VR) / mixed reality (MR) / augmented reality (AR) contexts. Moreover, while the term “head-mounted display (HMD)” is used herein according to some embodiments, some embodiments can be applied to wearable devices capable of, for example, XR, VR, AR, and / or MR, which can or can not be attached to a head.

[0163] An example method according to some embodiments can include encoding information describing a haptic experience, wherein the information describing the haptic experience includes one or more adaptation sets, wherein each of the one or more adaptation sets includes one or more representations corresponding to a haptic media track.

[0164] For some embodiments of the example method, each of the one or more representations corresponds to a same time period.

[0165] For some embodiments of the example method, the encoding of the information encodes the information in a container file.

[0166] For some embodiments of the example method, the one or more adaptation sets can include a primary haptic experience and a second haptic experience, wherein the second haptic experience corresponds to a first perceptual modality and a first channel.

[0167] For some embodiments of the example method, the adaptation set corresponding to the primary haptic experience can include initialization data corresponding to a haptic decoder.

[0168] For some embodiments of the example method, the adaptation set corresponding to the second haptic experience can include one or more segments of a corresponding haptic track.

[0169] For some embodiments of the example method, the adaptation set corresponding to the second haptic experience can include information identifying one or more segments of a corresponding haptic track.

[0170] For some embodiments of the example method, the first channel corresponds to a first frequency band.

[0171] For some embodiments of the example method, the information describing the haptic experience can further include information identifying one or more adaptation sets.

[0172] For some embodiments of the example method, the information describing the haptic experience can further include: information describing at least one available avatar for the haptic experience; and configuration information for at least one perception in the haptic experience, wherein the configuration information can include information describing one or more parallel haptic experience tracks.

[0173] For some embodiments of the example method, the information can further include information describing one or more haptic experience tracks, and the information describing the haptic experience track can further include: information describing at least one available avatar for the haptic experience; and configuration information for at least one perception in the haptic experience.

[0174] An example method / apparatus according to some embodiments can include: a processor; and a non-transitory computer-readable medium storing instructions that are operable, when executed by the processor, to cause the apparatus to: encode information describing a haptic experience, wherein the information describing the haptic experience includes one or more adaptation sets, and wherein each of the one or more adaptation sets includes one or more representations corresponding to a haptic media track.

[0175] An additional example method according to some embodiments can include: decoding information describing a haptic experience, wherein the information describing the haptic experience includes one or more adaptation sets, and wherein each of the one or more adaptation sets includes one or more representations corresponding to a haptic media track.

[0176] For some embodiments of the additional example method, each of the one or more representations corresponds to a same time period.

[0177] For some embodiments of the additional example method, the encoding of the information encodes the information in a container file.

[0178] For some embodiments of the additional example method, the one or more adaptation sets can include a primary haptic experience and a second haptic experience, and the second haptic experience corresponds to a first perception modality and a first channel.

[0179] For some embodiments of the additional example method, the adaptation set corresponding to the primary haptic experience can include initialization data corresponding to a haptic decoder.

[0180] For some embodiments of the additional example method, the adaptation set corresponding to the second haptic experience can include one or more segments of a corresponding haptic track.

[0181] For some embodiments of the additional example method, the adaptation set corresponding to the second haptic experience can include information identifying one or more segments of a corresponding haptic track.

[0182] Some embodiments of the additional example method can further include joining the initialization data with information identifying one or more segments from one or more adaptation sets to generate a bitstream.

[0183] Some embodiments of the additional example method can further include rendering the bitstream in a haptic experience environment.

[0184] For some embodiments of the additional example method, the first channel corresponds to a first frequency band.

[0185] For some embodiments of the additional example method, the information describing the haptic experience can further include information identifying one or more adaptation sets.

[0186] For some embodiments of the additional example method, the information describing the haptic experience can further include: information describing at least one available avatar for the haptic experience; and configuration information for at least one perception in the haptic experience, and the configuration information can include information describing one or more parallel haptic experience tracks.

[0187] For some embodiments of the additional example method, the information can further include information describing one or more haptic experience tracks, and the information describing the haptic experience track can further include: information describing at least one available avatar for the haptic experience; and configuration information for at least one perception in the haptic experience.

[0188] For some embodiments of the example method, the information describing the haptic experience and / or the encoded information describing the haptic experience is encoded according to the ISO Base Media File Format (ISOBMFF).

[0189] For some embodiments of the example method, the information describing the haptic experience and / or the encoded information describing the haptic experience is included in and streamed in a media file such as an MPEG-DASH (MPEG Dynamic Adaptive Streaming over HTTP) Media Presentation Descriptor (MPD) file.

[0190] For some embodiments of the example method, the information describing the haptic experience and / or the encoded information describing the haptic experience uses, is part of, and / or conforms to one or more of the MPEG standards ISO / IEC 23090-32 and ISO / IEC 23090-31.

[0191] An example method / apparatus according to some embodiments can include a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the apparatus to: decode information describing a haptic experience, wherein the information describing the haptic experience includes one or more adaptation sets, and wherein each of the one or more adaptation sets includes one or more representations corresponding to a haptic media track.

[0192] An example apparatus according to some embodiments can include at least one processor configured to perform any of the methods listed above.

[0193] An example apparatus according to some embodiments can include a computer- readable medium storing instructions for causing one or more processors to perform any of the methods listed above.

[0194] An example apparatus according to some embodiments can include at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform any of the methods listed above.

[0195] An example apparatus according to some embodiments can include a computer- readable medium storing a scene description file encoding information generated according to any of the methods listed above.

[0196] An example signal according to some embodiments can include a scene description file generated according to any of the methods listed above.

[0197] This disclosure describes a wide variety of aspects, including tools, features, embodiments, models, methods, and the like. Many of these aspects are described in detail, and often in a manner that can sound limiting, at least to show individual characteristics. However, this is for the purpose of clarity of description, and does not limit the disclosure or scope of the aspects. In fact, all of the different aspects can be combined and interchanged to provide additional aspects. Moreover, these aspects can also be combined and interchanged with aspects described in earlier filings.

[0198] The aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments are specifically illustrated, other embodiments are contemplated, and the discussion of a specific embodiment does not limit the breadth of the implementations. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting generated or encoded bitstreams. These and other aspects can be implemented as methods, apparatuses, computer-readable storage media having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media having stored thereon bitstreams generated according to any of the described methods.

[0199] In this disclosure, the terms “reconstruction” and “decoding” can be used interchangeably, the terms “pixel” and “sample” can be used interchangeably, and the terms “image”, “picture”, and “frame” can be used interchangeably. Typically, but not necessarily, the term “reconstruction” is used on the encoder side, while “decoding” is used on the decoder side.

[0200] The terms HDR (high dynamic range) and SDR (standard dynamic range) typically convey to the person of ordinary skill in the art specific values of the dynamic range. However, additional embodiments are also intended, in which a reference to HDR is understood to mean “higher dynamic range”, while a reference to SDR is understood to mean “lower dynamic range”. Such additional embodiments are not bound to any specific values of the dynamic range, which can often be associated with the terms “high dynamic range” and “standard dynamic range”.

[0201] Various methods are described herein, and each method includes one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions can be modified or combined. Additionally, terms such as “first”, “second”, etc. can be used to modify elements, components, steps, operations, etc. in various embodiments, such as for example “first decoding” and “second decoding”. Unless specifically required, the use of these terms does not imply a sequencing of the modified operations. Thus, in this example, the first decoding need not be performed before the second decoding, and can occur in a time period that precedes, during, or overlaps with the second decoding, for example.

[0202] For example, various numerical values can be used in this disclosure. The specific values are for the purpose of example, and the described aspects are not limited to these specific values.

[0203] Embodiments described herein can be implemented by computer software implemented by a processor or by other hardware, or by a combination of hardware and software. As a non-limiting example, embodiments can be implemented by one or more integrated circuits. As a non-limiting example, the processor can be any type of suitable processor adapted for the technical environment, and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on multi-core architecture.

[0204] Various implementations relate to decoding. As used in this disclosure, “decoding” can encompass, for example, all or part of the processing performed on a received encoded sequence in order to produce a final output suitable for display. In various embodiments, these processes include one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or instead include processes performed by decoders of various implementations described in this disclosure, such as extracting a picture from a tiled (packed) picture, determining an upsampling filter to use, and then upsampling the picture, and flipping the picture back to its intended orientation.

[0205] As further examples, in one embodiment, “decoding” refers only to entropy decoding, in another embodiment, “decoding” refers only to differential decoding, and in another embodiment, “decoding” refers to a combination of entropy decoding and differential decoding. Based on the specific context of the description, it will be clear whether the phrase “decoding process” is intended to refer specifically to a subset of operations or more broadly to the wider decoding process.

[0206] Various implementations relate to encoding. In a manner similar to the discussion above regarding “decoding,” as used in this disclosure, “encoding” can encompass, for example, all or part of the processes performed on an input video sequence in order to produce an encoded bitstream. In various embodiments, these processes include one or more processes typically performed by an encoder, such as partitioning, differential encoding, transform, quantization, and entropy encoding. In various embodiments, such processes also or instead include processes performed by encoders of various implementations described in this disclosure.

[0207] As further examples, in one embodiment, “encoding” refers only to entropy encoding, in another embodiment, “encoding” refers only to differential encoding, and in another embodiment, “encoding” refers to a combination of differential encoding and entropy encoding. Based on the specific context of the description, it will be clear whether the phrase “encoding process” is intended to refer specifically to a subset of operations or more broadly to the wider encoding process.

[0208] Various embodiments relate to rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is typically considered, often given a constraint on computational complexity. Rate-distortion optimization is typically formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solve the rate-distortion optimization problem. For example, these approaches can be based on extensive testing of all encoding options, including all considered mode or coding parameter values, and full evaluation of their coding cost and the associated distortion of the reconstructed signal after coding and decoding. Faster approaches can also be used to save encoding complexity, in particular based on predicting or approximating the distortion from the prediction or prediction residual signal instead of the reconstructed signal. A mix of these two approaches can also be used, such as by using the approximate distortion for only some of the possible encoding options, and the full distortion for other encoding options. Other approaches evaluate only a subset of the possible encoding options. More generally, many approaches employ any of a wide variety of techniques to perform the optimization, but the optimization is not necessarily a full evaluation of both the coding cost and the associated distortion.

[0209] When a diagram is presented as a flow chart, it will also be appreciated that one embodiment can be implemented as a block diagram, and vice versa. Similarly, where a diagram is presented as a block diagram, it will also be appreciated that one embodiment can be implemented as a flow chart, and vice versa.

[0210] Embodiments and aspects described herein can be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed can also be implemented in other forms (for example, an apparatus or program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.

[0211] Reference to “one embodiment” or “an embodiment” or “one implementation” or “an implementation”, as well as other variants thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment” or “in one implementation” or “in an implementation”, as well any other variant thereof, throughout this disclosure, are not necessarily all referring to the same embodiment.

[0212] Additionally, the present disclosure can relate to “determining” various pieces of information. Determining information can include one or more of, for example, estimating information, calculating information, predicting information, or retrieving information from memory.

[0213] Furthermore, the present disclosure can be directed to "accessing" various pieces of information. Accessing information can include, for example, one or more of receiving information, retrieving information (e.g., from storage), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0214] Furthermore, the present disclosure can be directed to "receiving" various pieces of information. As with "accessing," receiving is intended to be a broad term. Receiving information can include, for example, one or more of accessing information or retrieving information (e.g., from storage). Furthermore, "receiving" is typically involved in some fashion during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0215] It is to be appreciated that, for example, in the cases of "A / B," "A and / or B," and "at least one of A and B," any of the following is also expressly envisioned: only selection of the first listed option (A), or only selection of the second listed option (B), or only selection of both options (A and B). As another example, in the cases of "A, B, and / or C" and "at least one of A, B, and C," such phrasing is intended to cover the

[0216] Furthermore, as used herein, the word "signaling" relates to, among other things, indicating something to a corresponding decoder. For example, in certain embodiments, an encoder signals a particular one of a plurality of parameters for region-based filter parameter selection for de-artifact filtering. In this way, in embodiments, the same parameters are used at the encoder side and the decoder side. Thus, for example, the encoder can transmit (explicit signaling) a particular parameter to the decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter along with other parameters, then signaling can be used (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding transmission of any actual function, bit savings are realized in various embodiments. It is to be appreciated that signaling can be realized in a wide variety of ways. For example, in various embodiments, one or more syntax elements, flags, and the like are used to signal information to a corresponding decoder. While the foregoing relates to the verb form of the word "signaling," the word "signaling" can also be used as a noun herein.

[0217] Implementations can produce a variety of signals formatted to carry information such as can be stored or transmitted. The information can include, for example, instructions for performing a method, or data generated using one of the described implementations. For example, a signal can be formatted to carry a bitstream of a described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using a portion of the spectrum that is radiated, or that is wired, fiber-optic, or other communication medium). Formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. Signals can be transmitted through a variety of different wired and wireless links. Signals can be stored on processor-readable media.

[0218] A number of embodiments are described. Features of the embodiments can be provided individually or in any combination, across various claim categories and types. Further, embodiments can include one or more of the following features, apparatuses, or aspects, alone or in any combination, across various claim categories and types: • A bitstream or signal including one or more described syntax elements or variations thereof.

[0219] • A bitstream or signal including syntax conveying information generated according to any described embodiment.

[0220] • Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal including one or more described syntax elements or variations thereof.

[0221] • Creating and / or transmitting and / or receiving and / or decoding according to any described embodiment.

[0222] • A method, process, apparatus, medium storing instructions, medium storing data, or signal according to any described embodiment.

[0223] Note that various hardware elements of one or more described embodiments are referred to as "modules," which implement (i.e., perform, execute, etc.) various functions described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) that a person of ordinary skill in the relevant art recognizes as suitable for a given implementation. Each described module can also include instructions executable to implement one or more functions described as being implemented by the respective module, and it should be noted that these instructions can take the form of, or include, hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., and can be stored in any suitable non-transitory computer readable medium or media, such as generally referred to as RAM, ROM, etc.

[0224] Although features and elements are described above in particular combinations, one of ordinary skill in the art will appreciate that each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein can be implemented in a computer program, software, or firmware incorporated in a computer readable medium for execution by a computer or processor. Examples of computer readable storage media include, but are not limited to, read only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs). A processor in association with software can be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method comprising: Encoding information that describes the tactile experience, The information describing the tactile experience includes one or more adapter sets, and Each of the one or more adapter sets includes one or more representations corresponding to haptic media tracks.

2. The method of claim 1, wherein each of the one or more representations corresponds to the same time period.

3. The method according to any one of claims 1-2, wherein the encoding of the information encodes the information in a container file.

4. The method according to any one of claims 1-3, The one or more adaptation sets mentioned above include primary haptic experience and secondary haptic experience, and The second tactile experience corresponds to the first sensory modality and the first channel.

5. The method according to claim 4, The adaptation set corresponding to the primary tactile experience includes initialization data corresponding to the tactile decoder.

6. The method according to any one of claims 4-5, The adaptation set corresponding to the second tactile experience includes one or more segments of the corresponding tactile track.

7. The method according to claim 6, The adaptation set corresponding to the second tactile experience includes information identifying one or more segments of the corresponding tactile track.

8. The method according to any one of claims 4-7, wherein the first channel corresponds to the first frequency band.

9. The method according to any one of claims 1-8, wherein the information describing the tactile experience further includes information identifying the one or more adapter sets.

10. The method according to any one of claims 1-9, The information describing the tactile experience further includes: Information describing at least one available avatar for the tactile experience; as well as Configuration information for at least one perception in the tactile experience. The configuration information includes information describing one or more parallel haptic experience tracks.

11. The method according to any one of claims 1-9, The information further includes information describing one or more tactile experience tracks, and The information describing the tactile experience track further includes: Information describing at least one available avatar for tactile experience; as well as Configuration information for at least one perception in the tactile experience.

12. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to: Encoding information that describes the tactile experience, The information describing the tactile experience includes one or more adapter sets, and Each of the one or more adapter sets includes one or more representations corresponding to haptic media tracks.

13. A method comprising: Decoding information describing tactile experience The information describing the tactile experience includes one or more adapter sets, and Each of the one or more adapter sets includes one or more representations corresponding to haptic media tracks.

14. The method of claim 13, wherein each of the one or more representations corresponds to the same time period.

15. The method according to any one of claims 13-14, wherein the encoding of the information encodes the information in a container file.

16. The method according to any one of claims 13-15, The one or more adaptation sets mentioned above include primary haptic experience and secondary haptic experience, and The second tactile experience corresponds to the first sensory modality and the first channel.

17. The method according to claim 16, The adaptation set corresponding to the primary tactile experience includes initialization data corresponding to the tactile decoder.

18. The method according to claim 17, The adaptation set corresponding to the second tactile experience includes one or more segments of the corresponding tactile track.

19. The method according to claim 18, The adaptation set corresponding to the second tactile experience includes information identifying one or more segments of the corresponding tactile track.

20. The method of claim 19, further comprising: The initialization data is concatenated with information from one or more adapter sets that identifies the one or more segments to generate a bitstream.

21. The method of claim 20, further comprising rendering the bitstream in a haptic experience environment.

22. The method according to any one of claims 16-21, wherein the first channel corresponds to the first frequency band.

23. The method according to any one of claims 13-22, wherein the information describing the tactile experience further includes information identifying the one or more adapter sets.

24. The method according to any one of claims 13-23, The information describing the tactile experience further includes: Information describing at least one available avatar for the tactile experience; as well as Configuration information for at least one perception in the tactile experience. The configuration information includes information describing one or more parallel haptic experience tracks.

25. The method according to any one of claims 13-23, The information further includes information describing one or more tactile experience tracks, and The information describing the tactile experience track further includes: Information describing at least one available avatar for tactile experience; as well as Configuration information for at least one perception in the tactile experience.

26. The method according to any one of claims 1-25, wherein the information describing the tactile experience, and / or the encoded information describing the tactile experience, is encoded according to the ISO Basic Media File Format (ISOBMFF).

27. The method according to any one of claims 1-25, wherein the information describing the tactile experience and / or the encoded information describing the tactile experience are included in and streamed in a media file such as an MPEG-DASH (MPEG Dynamic Adaptive Streaming over HTTP) media presentation descriptor (MPD) file.

28. The method according to any one of claims 1-25, wherein the information describing the tactile experience and / or the encoded information describing the tactile experience uses, is a part thereof and / or conforms to one or more of the MPEG standards ISO / IEC 23090-32 and ISO / IEC 23090-31.

29. An apparatus comprising: processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by the processor, are operable to cause the device to: Decoding information describing tactile experience The information describing the tactile experience includes one or more adapter sets, and Each of the one or more adapter sets includes one or more representations corresponding to haptic media tracks.

30. An apparatus comprising at least one processor configured to perform the method according to any one of claims 1-29.

31. An apparatus comprising a computer-readable medium storing instructions for causing one or more processors to perform the method according to any one of claims 1-29.

32. An apparatus comprising at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform the method according to any one of claims 1-29.

33. A computer-readable medium storing a scene description file of encoded information generated according to any one of claims 16-29.

34. A signal comprising a scene description file generated according to any one of claims 1-29.

Images