Dash signaling for adaptive streaming of haptic media

Abstract

Some embodiments of the method include encoding information describing a haptic experience, the information describing the haptic experience may include one or more adaptation sets, each of which may include one or more representations corresponding to a haptic media track. Some embodiments of the method also include decoding information describing a haptic experience, the information describing the haptic experience may include one or more adaptation sets, each of which may include one or more representations corresponding to a haptic media track.

Description

Background Art

[0001] A haptic sequence is a set of data encoded for rendering based on haptics and positioning in space, just as a video sequence is a set of data encoded for rendering using vision. A haptic sequence encodes transient data, represented, for example, as a track associated with a haptic device. A haptic device may render separate modalities related to haptics and positioning in space, such as vibration, force, position, velocity, or temperature.

[0002] Cross-references to related applications This application claims the benefit of U.S. Patent Application No. 63 / 526,954, filed Jul. 14, 2023, entitled “DASH SIGNALING FOR ADAPTIVE STREAMING OF HAPTICS MEDIA,” and is hereby incorporated by reference in its entirety.

[0003] Incorporation by reference This application incorporates by reference in its entirety the following application, U.S. Provisional Patent Application No. 63 / 417,638, filed Oct. 19, 2022, entitled “Carriage of Coded Haptics Data in Media Containers” (the “’63B application”).

Summary of the Invention

[0004] Embodiments described herein include methods used for video encoding and decoding (collectively “encoding”).

[0005] An exemplary method, according to several embodiments, includes encoding information describing a haptic experience, the information describing the haptic experience may include one or more sets of adaptations, each of which may include one or more representations corresponding to a haptic media track.

[0006] With respect to some embodiments of the exemplary method, each of one or more expressions corresponds to the same period of time.

[0007] In some embodiments of the exemplary method, information encoding involves encoding the information in a container file.

[0008] In some embodiments of the exemplary method, one or more adaptation sets may include a main haptics experience and a second haptics experience, the second haptics experience corresponding to a first perception modality and a first channel.

[0009] In some embodiments of the exemplary method, the adaptation set corresponding to the main haptics experience may include initialization data corresponding to the haptics decoder.

[0010] With respect to some embodiments of the exemplary method, the adaptation set corresponding to the second haptic experience may include one or more fragments of the corresponding haptic track.

[0011] In some embodiments of the exemplary method, the adaptation set corresponding to the second haptic experience may include information identifying one or more fragments of the corresponding haptic track.

[0012] In some embodiments of the exemplary method, the first channel corresponds to a first frequency band.

[0013] With respect to some exemplary embodiments of the method, the information describing the haptic experience may further include information identifying one or more sets of adaptations.

[0014] With respect to some embodiments of the exemplary method, the information describing the haptic experience may further include information describing at least one available avatar for the haptic experience, and configuration information for at least one perception in the haptic experience, the configuration information may include information describing one or more parallel haptic experience tracks.

[0015] With respect to some embodiments of the exemplary method, the information may further include information describing one or more haptic experience tracks, the information describing the haptic experience tracks may further include information describing at least one available avatar for the haptic experience, and configuration information for at least one perception in the haptic experience.

[0016] An exemplary method / apparatus, according to several embodiments, includes a processor and a non-temporary computer-readable medium that stores instructions operable to cause the apparatus to encode information describing a haptic experience, the information describing a haptic experience may include one or more sets of adaptations, each of which includes one or more representations corresponding to a haptic media track.

[0017] An exemplary additional method, according to several embodiments, includes decoding information describing a haptic experience, which may include one or more adaptation sets, each of which includes one or more representations corresponding to a haptic media track.

[0018] With respect to some embodiments of the additional exemplary methods, each of one or more expressions corresponds to the same period of time.

[0019] Regarding some embodiments of additional exemplary methods, information encoding involves encoding information in a container file.

[0020] With respect to some embodiments of additional exemplary methods, one or more adaptation sets may include a main haptic experience and a second haptic experience, the second haptic experience corresponding to a first perception modality and a first channel.

[0021] In some embodiments of additional exemplary methods, the adaptation set corresponding to the main haptics experience may include initialization data corresponding to the haptics decoder.

[0022] With respect to some embodiments of additional exemplary methods, the adaptation set corresponding to the second haptic experience may include one or more fragments of the corresponding haptic track.

[0023] With respect to some embodiments of additional exemplary methods, the adaptation set corresponding to a second haptic experience may include information identifying one or more fragments of the corresponding haptic track.

[0024] Some embodiments of the additional exemplary method may further include concatenating initialization data with information identifying one or more adaptation sets to generate a bitstream.

[0025] Some embodiments of the additional exemplary method may further include rendering the bitstream in a haptic experience environment.

[0026] Regarding some embodiments of the additional exemplary method, the first channel corresponds to a first frequency band. Regarding some embodiments of the additional exemplary method, the information describing the haptic experience may further include information identifying one or more adaptation sets.

[0027] Regarding some embodiments of the additional exemplary method, the information describing the haptic experience may further include information describing at least one available avatar for the haptic experience and configuration information for at least one perception in the haptic experience, and the configuration information may include information describing one or more parallel haptic experience tracks.

[0028] Regarding some embodiments of the additional additional exemplary method, the information may further include information describing one or more haptic experience tracks, and the information describing the haptic experience tracks may further include information describing at least one available avatar for the haptic experience and configuration information for at least one perception in the haptic experience.

[0029] For some embodiments of the exemplary method, information describing the haptics experience, and / or the encoded information describing the haptics experience, is encoded according to ISOBMFF (ISO Base Media File Format).

[0030] For some embodiments of the exemplary method, information describing the haptics experience, and / or the encoded information describing the haptics experience, is included in a media file, such as an MPD (media presentation descriptor) file of MPEG-DASH (MPEG Dynamic Adaptive Streaming Over HTTP), and is streamed.

[0031] For some embodiments of the exemplary method, information describing the haptics experience, and / or the encoded information describing the haptics experience, is one or more parts of, and / or compliant with, one or more of MPEG standards ISO / IEC 23090-32 and ISO / IEC 23090-31.

[0032] An exemplary method / apparatus according to some embodiments includes a processor and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the apparatus to decode information describing the haptics experience, where the information describing the haptics experience may include one or more adaptation sets, and each of the one or more adaptation sets includes one or more representations corresponding to a haptics media track.

[0033] An exemplary apparatus according to some embodiments may include at least one processor configured to perform any one of the methods listed above. Exemplary devices according to some embodiments may include a computer-readable medium that stores instructions for causing one or more processors to perform any one of the methods listed above.

[0034] Exemplary devices according to some embodiments may include at least one processor and at least one non-temporary computer-readable medium for storing instructions for causing the at least one processor to perform one of the methods listed above.

[0035] Exemplary devices according to several embodiments may include a computer-readable medium for storing scene description files of encoded information generated according to any one of the methods listed above.

[0036] An exemplary signal according to several embodiments may include a scene description file generated according to one of the methods listed above.

[0037] In additional embodiments, an encoder and decoder device is provided for performing the methods described herein. The encoder or decoder device may include a processor configured to perform the methods described herein. The device may include a computer-readable medium (e.g., a non-temporary medium) for storing instructions in order to perform the methods described herein. In some embodiments, the computer-readable medium (e.g., a non-temporary medium) stores video encoded using one of the methods described herein.

[0038] Furthermore, one or more of these embodiments also provide a computer-readable recording medium storing instructions for encoding or decoding video data to perform a bidirectional optical flow according to one of the methods described above. Furthermore, these embodiments also provide a computer-readable recording medium storing a bitstream generated according to one of the methods described above. Furthermore, these embodiments also provide a method and apparatus for transmitting a bitstream generated according to one of the methods described above. Furthermore, these embodiments also provide a computer program product containing instructions for performing one of the methods described above. [Brief explanation of the drawing]

[0039] [Figure 1A] This is a system diagram illustrating an exemplary communication system according to several embodiments. [Figure 1B] This is a system diagram illustrating an exemplary wireless transmit / receive unit (WTRU) that may be used in a communication system as illustrated in Figure 1A, according to several embodiments. [Figure 1C] This is a system diagram illustrating an exemplary set of interfaces for a system according to several embodiments. [Figure 2] This system diagram illustrates an exemplary set of interfaces for an MPEG-I node hierarchy that supports elements of scene interactivity according to several embodiments. [Figure 3] This is a system diagram illustrating an exemplary set of interfaces for an MPEG haptic architecture according to several embodiments. [Figure 4] This is a system diagram illustrating an exemplary hierarchical data structure for two codec formats according to several embodiments. [Figure 5] This is a schematic example illustrating an exemplary NAL unit structure in a haptic bitstream according to several embodiments. [Figure 6]This is a schematic example illustrating exemplary NAL unit payload types according to several embodiments. [Figure 7] This is a code listing illustrating exemplary EditListBox class structures according to several embodiments. [Figure 8] This is a schematic example illustrating an exemplary MPD hierarchical data model according to several embodiments. [Figure 9] This is a system diagram illustrating an exemplary DASH configuration for grouping adaptation sets according to several embodiments. [Figure 10] This is a code listing illustrating exemplary XML schemas according to several embodiments. Figure 9 shows exemplary data types for various elements and attributes related to the XML schema. [Figure 11] This is a code listing illustrating exemplary XML schemas according to several embodiments. [Figure 12] This flowchart illustrates exemplary processes for encoding haptic data according to several embodiments. [Figure 13] This flowchart illustrates exemplary processes for decoding haptic data according to several embodiments. [Modes for carrying out the invention]

[0040] Entities, connections, arrangements, and similar things depicted in and described in relation to various drawings are given as examples, not as limitations. As stated above, any and all statements or indications relating to something that a particular drawing "depicts," something that "is" or "has" a particular element or entity in a particular drawing, which may be read absolute and therefore limiting, both on their own and out of context, may only be read appropriately as being constructively preceded by a clause such as, for example, "In at least one embodiment, ...". For the sake of brevity and clarity of expression, the implicit preceding clauses mentioned above are not to be unpleasantly repeated in the detailed descriptions.

[0041] Figure 1A illustrates an exemplary communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple-access system that provides content such as voice, data, video, messaging, and broadcast to multiple wireless users. The communication system 100 may enable multiple wireless users to access the above content through the sharing of system resources, including wireless bandwidth. For example, the communication system 100 may employ one or more channel access methods such as CDMA (Code Division Multiple Access), TDMA (Time Division Multiple Access), FDMA (Frequency Division Multiple Access), OFDMA (Orthogonal Frequency Division Multiple Access), SC-FDMA (Single Carrier FDMA), ZT UW DTS-s OFDM (zero-tail unique-word DFT-Spread OFDM), UW-OFDM (unique word OFDM), resource block-filtered OFDM, and FBMC (filter bank multicarrier).

[0042] As shown in Figure 1A, the communication system 100 may include WTRUs (Wireless Transmitter / Receiver Units) 102a, 102b, 102c, 102d, RAN 104, CN 106, Public Switched Telephone Network (PSTN) 108, the Internet 110, and other networks 112, but the disclosed embodiments will be understood to anticipate several WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, and 102d may be any type of device configured to operate and / or communicate in a wireless environment. For example, WTRU102a, 102b, 102c, and 102d may all be referred to as “station” and / or “STA,” and may be configured to transmit and / or receive wireless signals, including UEs (User Equipment), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, cellular phones, PDAs (Personal Digital Assistants), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, IoT (Internet of Things) devices, watches or other wearables, HMDs (Head-Mounted Displays), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in the context of industrial and / or automated processing chains), consumer electronics devices, and devices operating on commercial and / or industrial wireless networks. Any of WTRU102a, 102b, 102c, and 102d may be interchangeable with UE.

[0043] Furthermore, the communication system 100 may also include base stations 114a and / or base stations 114b. Each of the base stations 114a and 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, and 102d to facilitate access to one or more communication networks, such as CN 106, the Internet 110, and / or other networks 112. For example, base stations 114a and 114b may be a BTS (Radio Base Station Equipment), Node-B, eNode B, Home Node B, Home eNode B, gNB, NR Node B, Site Controller, AP (Access Point), Wireless Router, etc. While base stations 114a and 114b are each depicted as single elements, it will be understood that base stations 114a and 114b may include any number of interconnected base stations and / or network elements.

[0044] Base station 114a may also be part of RAN 104, which may include other base stations and / or network elements (not shown), such as BSC (Base Station Control Unit), RNC (Radio Network Control Unit), and relay nodes. Base station 114a and / or base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies, which may be called a cell (not shown). The frequencies mentioned may be permitted spectrum, unpermitted spectrum, or a combination of permitted and unpermitted spectrum. A cell may provide coverage to a particular geographic area that may be relatively fixed or may change in the future. Furthermore, a cell may be divided into sectors. For example, a cell associated with base station 114a may be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, i.e., one for each sector of the cell. In the embodiment, the base station 114a may employ MIMO (multiple-input multiple output) technology and utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.

[0045] Base stations 114a and 114b may communicate with one or more WTRUs 102a, 102b, 102c, and 102d via an air interface 116, which may be any suitable wireless communication link (e.g., RF (radio frequency), microwave, centimeter wave, micrometer wave, IR (infrared), UV (ultraviolet), visible light, etc.). The air interface 116 may be established using any suitable RAT (radio access technology).

[0046] More specifically, as described above, the communication system 100 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, base stations 114a and WTRUs 102a, 102b, and 102c in RAN 104 may implement radio technologies such as UTRA (UMTS (Universal Mobile Telecommunications System) Terrestrial Radio Access) which may establish an air interface 116 using WCDMA (wideband CDMA). WCDMA may include communication protocols such as HSPA (High-Speed ​​Packet Access) and / or HSPA+ (Evolved HSPA). HSPA may include HSDPA (High-Speed ​​DL (Downlink) Packet Access) and / or HSUPA (High-Speed ​​UL Packet Access).

[0047] In the embodiment, base stations 114a and WTRUs 102a, 102b, and 102c may implement radio technologies such as E-UTRA (Evolved UMTS Terrestrial Radio Access), which may establish an air interface 116 using, for example, LTE (Long Term Evolution) and / or LTE-A (LTE-Advanced) and / or LTE-A Pro (LTE-Advanced Pro).

[0048] In the embodiment, base stations 114a and WTRUs 102a, 102b, and 102c may implement radio technologies such as radio access for NR (New Radio), for example, to establish an air interface 116 using NR.

[0049] In the embodiment, base stations 114a and WTRUs 102a, 102b, and 102c may implement multiple radio access technologies. For example, base stations 114a and WTRUs 102a, 102b, and 102c may implement both LTE radio access and NR radio access using, for example, the DC (dual connectivity) principle. Thus, the air interface used by WTRUs 102a, 102b, and 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to and from multiple types of base stations (e.g., eNBs and gNBs).

[0050] In other embodiments, base stations 114a and WTRUs 102a, 102b, 102c may implement wireless technologies such as IEEE 802.11 (i.e., WiFi (Wireless Fidelity)), IEEE 802.16 (i.e., WiMAX (Worldwide Interoperability for Microwave Access)), CDMA2000, CDMA2000 1X, CDMA2000EV-DO, IS-2000 (Interim Standard 2000), IS-95 (Interim Standard 95), IS-856 (Interim Standard 856), GSM (Global System for Mobile communications), EDGE (Enhanced Data rates for GSM Evolution), GERAN (GSM EDGE), and similar technologies.

[0051] In Figure 1A, base station 114b may be, for example, a wireless router, home Node B, home eNode B, or access point, and may utilize any RAT suitable for facilitating wireless connectivity in localized areas such as businesses, homes, vehicles, campuses, industrial facilities, aerial walkways (e.g., for drone use), and roadways. In one embodiment, base station 114b and WTRU 102c, 102d may implement wireless technology such as IEEE 802.11 to establish a WLAN (wireless local area network). In another embodiment, base station 114b and WTRU 102c, 102d may implement wireless technology such as IEEE 802.15 to establish a WPAN (wireless personal area network). In yet another embodiment, base stations 114b and WTRUs 102c, 102d may utilize cellular-based RATs (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.) to establish picocells or femtocells. As shown in Figure 1A, base station 114b may have a direct connection to the internet 110. Therefore, base station 114b may not be required to access the internet 110 via CN 106.

[0052] RAN104 may be in communication with CN106 and may be any type of network configured to provide voice, data, applications, and / or VoIP (voice over internet protocol) services to one or more of WTRU102a, 102b, 102c, and 102d. The data may have various QoS (quality of service) requirements, such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, and mobility requirements. CN106 may provide call control, billing services, mobile location-based services, prepaid calling, internet connectivity, video distribution, and / or high-level security functions, such as user authentication. Although not shown in Figure 1A, it will be understood that RAN104 and / or CN106 may be in direct or indirect communication with other RANs employing the same RAT as RAN104 or different RATs. For example, in addition to being connected to RAN104, which may be using NR wireless technology, CN106 may also be in communication with another RAN (not shown) employing GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi wireless technology.

[0053] Furthermore, CN106 may also serve as a gateway for WTRU102a, 102b, 102c, and 102d to access PSTN108, the Internet 110, and / or other networks 112. PSTN108 may include a circuit-switched telephone network providing POTS (plain old telephone service). The Internet 110 may include a global system of interconnected computer networks and devices using common communication protocols such as TCP (transmission control protocol), UDP (user datagram protocol), and / or IP in the TCP / IP (Internet Protocol) suite. Network 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, network 112 may include another CN connected to one or more RANs that may employ the same RAT as RAN104 or a different RAT.

[0054] Some or all of the WTRUs 102a, 102b, 102c, and 102d in the communication system 100 may include multimode capability (for example, WTRUs 102a, 102b, 102c, and 102d may include multiple transceivers to communicate with separate wireless networks via separate wireless links). For example, WTRU 102c, shown in Figure 1A, may be configured to communicate with base station 114a, which may employ cellular-based radio technology, and base station 114b, which may employ IEEE 802 radio technology.

[0055] Figure 1B is a system diagram illustrating an exemplary WTRU 102. As shown in Figure 1B, the WTRU 102 may include, among others, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, a non-removable memory 130, a removable memory 132, a power supply 134, a GPS (Global Positioning System) chipset 136, and / or other peripherals 138. It will be understood that the WTRU 102 may include any sub-combination of the elements described above, without inconsistency with the embodiment.

[0056] The processor 118 may be a general-purpose processor, a dedicated processor, a conventional processor, a DSP (digital signal processor), multiple microprocessors, one or more microprocessors with a DSP core, a controller, a microcontroller, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) circuit, any other type of IC (integrated circuit), a state machine, and the like. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to a transceiver 120, which may be coupled to a transmit / receive element 122. Figure 1B depicts the processor 118 and the transceiver 120 as separate components, while it will be understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0057] The transmit / receive element 122 may be configured to transmit or receive signals to or from a base station (e.g., base station 114a) via the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In an embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive, for example, IR signals, UV signals, or visible light signals. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF and optical signals. It will be understood that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0058] Although the transmit / receive element 122 is depicted as a single element in Figure 1B, the WTRU 102 may contain any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Therefore, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) to transmit and receive wireless signals via the air interface 116.

[0059] The transceiver 120 may be configured to modulate the signal that is to be transmitted by the transmit / receive element 122, and to demodulate the signal that is to be received by the transmit / receive element 122. As mentioned above, the WTRU 102 may have multimode capabilities. Therefore, for example, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate by multiple RATs, such as NR and IEEE 802.11.

[0060] The processor 118 of the WTRU102 may be connected to and receive user input data via a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., an LCD (liquid crystal display) display unit or an OLED (organic light-emitting diode) display unit). Furthermore, the processor 118 may output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information and store data in any suitable type of memory, such as a non-removable memory 130 and / or removable memory 132. The non-removable memory 130 may include RAM (random-access memory), ROM (read-only memory), a hard disk, or any other type of memory storage device. The removable memory 132 may include a SIM (subscriber identity module) card, a Memory Stick, an SD (secure digital) memory card, and similar. In other embodiments, the processor 118 may access information and store data in memory, for example, a server or home computer (not shown), which is not physically located in the WTRU 102.

[0061] The processor 118 may receive power from a power supply 134 and may be configured to distribute and / or control power to other components in the WTRU 102. The power supply 134 may be any device suitable for supplying power to the WTRU 102. For example, the power supply 134 may include one or more dry cell batteries (e.g., NiCd (nickel-cadmium), NiZn (nickel-zinc), NiMH (nickel-metal hydride), Li-ion (lithium-ion), etc.), solar cells, fuel cells, and similar devices.

[0062] Furthermore, the processor 118 may be coupled to a GPS chipset 136 which may be configured to provide location information (e.g., longitude and latitude) about the current location of the WTRU 102. In addition to or instead of the information from the GPS chipset 136, the WTRU 102 may determine its location based on receiving location information from base stations (e.g., base stations 114a, 114b) via the air interface 116 and / or based on the timing of signals received from two or more neighboring base stations. It will be understood that the WTRU 102 may acquire location information through any appropriate location-determination method, without being inconsistent with the embodiments.

[0063] Furthermore, the processor 118 may be coupled to other peripherals 138, which may include one or more software modules and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, peripherals 138 may include an accelerometer, e-compass, satellite transceiver, digital camera (for photos and / or video), USB (Universal Serial Bus) port, vibration device, television transceiver, hands-free headset, Bluetooth® module, FM (freqUEncy modulated) radio unit, digital music player, media player, video game player module, internet browser, VR / AR (virtual reality and / or augmented reality) device, activity tracker, and the like. The peripheral device 138 may include one or more sensors, which may be one or more of the following: a gyroscope, accelerometer, Hall effect sensor, magnetometer, compass sensor, proximity sensor, temperature sensor, time sensor, geolocation sensor, altimeter, light sensor, touch sensor, magnetometer, barometer, gesture sensor, biometric sensor, and / or humidity sensor.

[0064] WTRU102 may include a full-duplex radio where some or all of the transmission and reception of signals (e.g., associated with a particular subframe with respect to both UL (e.g., for transmission) and downlink (e.g., for reception) may be in parallel and / or simultaneous. The full-duplex radio may include an interference management unit to reduce and / or substantially eliminate self-interference by either hardware (e.g., chokes) or signal processing by a processor (e.g., separate processors (not shown) or by processor 118). In embodiments, WRTU102 may include a half-duplex radio where some or all of the transmission and reception of signals (e.g., associated with a particular subframe with respect to either UL (e.g., for transmission) or downlink (e.g., for reception) may be in parallel and / or simultaneously.

[0065] Although the WTRU is described as a wireless terminal in Figures 1A-1B, in a typical embodiment, the terminal may use a wired communication interface with a communication network (for example, temporarily or permanently).

[0066] In a typical embodiment, the other network 112 may be a WLAN.

[0067] From the perspective of Figures 1A-1B and the corresponding descriptions, one or more or all of the functions described herein may be performed by one or more emulation devices (not shown). An emulation device may be one or more devices configured to emulate one or more or all of the functions described herein. For example, an emulation device may be used to test other devices and / or to simulate network and / or WTRU functions.

[0068] Emulation devices may be designed to implement one or more tests of other devices in a lab environment and / or an operator's network environment. For example, one or more emulation devices may perform one, more, or all functions while being implemented and / or deployed as part of a wired and / or wireless communication network, either entirely or partially, to test other devices in a communication network. One or more emulation devices may perform one, more, or all functions while being temporarily implemented and / or deployed as part of a wired and / or wireless communication network. Emulation devices may be directly coupled to another device for testing purposes and / or testing may be performed using over-the-air (OTA) wireless communication.

[0069] One or more emulation devices may perform one or more functions, including all of the above, while not implemented / deployed as part of a wired and / or wireless communication network. For example, an emulation device may be used in a testing scenario in a testing laboratory and / or a wired and / or wireless communication network that is not deployed (e.g., for testing) to implement testing of one or more components. One or more emulation devices may be test equipment. Direct RF coupling and / or wireless communication via RF circuitry (which may include, for example, one or more antennas) may be used by emulation devices to transmit and / or receive data.

[0070] The embodiments described herein are not limited to being implemented in a WTRU. The above embodiments may be implemented using other systems, such as the system shown in Figure 1C.

[0071] Figure 1C is a system diagram illustrating an exemplary set of interfaces for a system according to several embodiments. The extended reality display device may be implemented using a system such as the system in Figure 1C, together with control electronics. System 140 can be embodied as a device including various components described below and configured to perform one or more embodiments described in this document. Examples of the above device include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home electrical appliances, and servers. The elements of System 140 can be embodied individually or in combination as a single integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, the processing and encoder / decoder elements of System 140 are distributed across multiple ICs and / or separate components. In various embodiments, System 140 is communicated to one or more other systems or other electronic devices, for example, via a communication bus or via dedicated input and / or output ports. In various embodiments, the system 140 is configured to implement one or more of the embodiments described in this document.

[0072] System 140 includes, for example, at least one processor 142 configured to execute instructions loaded to implement various embodiments described in this document. Processor 142 may include embedded memory, input / output interfaces, and various other circuits as known in the Art. System 140 includes at least one memory 144 (e.g., a volatile memory device and / or a non-volatile memory device). System 140 may include a storage device 148 which may include non-volatile memory and / or volatile memory, including, but not limited to, EEPROM (Electrically Erasable Programmable Read-Only Memory), ROM (Read-Only Memory), PROM (Programmable Read-Only Memory), RAM (Random Access Memory), DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), flash, magnetic disk drives, and / or optical disk drives. Storage device 148 may, in non-limiting examples, include internal storage devices, attached storage devices (including detachable and non-detachable storage devices), and / or network-accessible storage devices.

[0073] System 140 includes, for example, an encoder / decoder module 146 configured to process data that provides encoded or decoded video, the encoder / decoder module 146 may include its own processor and memory. Encoder / decoder module 146 represents a module(s) that can be included in a device that performs encoding and / or decoding functions. As is known, the device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 146 may be implemented as a separate element of system 140, or it may be incorporated into processor 142 as a combination of hardware and software, as is known to those skilled in the art.

[0074] Program code loaded onto a processor 142 or encoder / decoder 146 performing various actions as described herein can be stored in a storage device 148 and subsequently loaded into memory 144 for execution by the processor 142. According to various embodiments, one or more processors 142, memory 144, storage device 148, and encoder / decoder modules 146 can store one or more items during the execution of the processes described herein. The stored items may include, but are not limited to, input video, decoded video, or a portion of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0075] In some embodiments, internal memory of the processor 142 and / or encoder / decoder module 146 is used to store instructions and provide working memory for the processing required during encoding or decoding. However, in other embodiments, external memory of the processing device (for example, the processing device can be either the processor 142 or the encoder / decoder module 142) is used for one or more of the functions described above. The external memory can be memory 144 and / or storage device 148, for example, dynamic volatile memory and / or non-volatile flash memory. In some embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, a high-speed external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as MPEG-2 (MPEG refers to the Moving Picture Experts Group, and MPEG-2 is further called ISO / IEC 13818, and 13818-1 is known as H.222, and 13818-2 is known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, JVET, a standard developed by the Joint Video Experts Team).

[0076] Inputs to the elements of System 140 can be provided via various input devices, as shown in Block 162. These input devices include, but are not limited to, (i) an RF section for receiving, for example, RF (radio frequency) signals transmitted by a broadcasting station via radio waves, (ii) a COMP (Component) input terminal (or a set of COMP input terminals), (iii) a USB (Universal Serial Bus) input terminal, and / or (iv) an HDMI (High Definition Multimedia Interface) input terminal. Other examples not shown in Figure 1A include composite video.

[0077] In various embodiments, the input device of block 162 associates with various input processing elements known in the Art. For example, the RF portion can be associated with elements suitable for (i) selecting a desired frequency (also said to select a signal or band-limit a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting a signal frequency band, which may be called a channel in some embodiments, to a narrower band of the selecting frequency, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream for data packets. The RF portion of various embodiments includes one or more elements that perform the functions described above, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error collectors, and demultiplexers. The RF portion can include tuners that perform various functions, such as down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency close to the baseband) or to the baseband. In one embodiment of a set-top box, the RF section and associated input processing elements receive, filter, down-convert, and filter again to a desired frequency band, thereby performing frequency selection. Various embodiments rearrange the order of the elements described above (and others), remove some of the elements mentioned, and / or add other elements that perform similar or different functions. Adding elements can include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0078] Additionally, USB and / or HDMI terminals may include their respective interface processors to connect system 140 to other electronic devices via USB and / or HDMI. It is understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or, if necessary, in processor 142. Similarly, aspects of USB or HDMI interface processing may be implemented in a separate input processing IC or, if necessary, in processor 142. Demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, for example, processor 142 and encoder / decoder 146, which works in cooperation with memory and storage elements to process the data stream for submission to output devices as necessary.

[0079] The various elements of system 140 can be provided within an integrated housing, where the various elements are interconnected and can transmit data between them using an internal bus known in the art, such as an I2C (Inter-IC) bus, wiring, and printed circuit board, which is suitable for a linked array 164.

[0080] System 140 includes a communication interface 150 that enables communication with other devices via a communication channel 152. The communication interface 150 may, but is not limited to, include a transceiver configured to transmit and receive data via the communication channel 152. The communication interface 150 may, but is not limited to, include a modem or a network card, and the communication channel 152 may be implemented, for example, in a wired and / or wireless medium.

[0081] In various embodiments, data is streamed to system 140 or otherwise provided, for example, using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in the embodiments described above is received via a communication channel 152 and a communication interface 150, which are modified to accommodate Wi-Fi communication. Typically, the communication channel 152 in the embodiments described above is coupled to an access point or router that provides access to an external network, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, the streamed data is provided to system 140 using a set-top box that distributes the data via an HDMI connection in input block 162. Still, in other embodiments, the streamed data is provided to system 140 using an RF connection in input block 162. As shown above, various embodiments provide data in ways other than streaming. In addition, various embodiments use wireless networks other than Wi-Fi, e.g., cellular networks or Bluetooth networks.

[0082] The system 140 can provide output signals to various output devices, including a display 166, a speaker 168, and other peripheral devices 170. In various embodiments, the display 166 includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 166 can be for a television, tablet, laptop, cellular phone (mobile phone), or other device. Furthermore, the display 166 can be integrated with other components (e.g., in a smartphone) or separate (e.g., with an external monitor for a laptop). In various examples of embodiments, the other peripheral devices 170 include one or more of a standalone digital video disc (or digital versatile disc) (DVR for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments utilize one or more peripheral devices 170 that provide functions based on the output of the system 140. For example, a disc player plays the role of playing the output of the system 140.

[0083] In various embodiments, control signals are communicated between the system 140 and the display 166, speaker 168, or other peripheral devices 170 using signaling such as AV.Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. Output devices can be connected to the system 140 via dedicated connections through their respective interfaces 154, 156, and 158. Alternatively, output devices can be connected to the system 140 via communication interface 150 using communication channel 152. The display 166 and speaker 168 can be integrated into a single unit with other components of the system 140, for example, in an electronic device such as a television. In various embodiments, the display interface 154 includes a display driver, such as a timing controller (TCon) chip.

[0084] Alternatively, the display 166 and speaker 168 can be isolated from one or more other components, for example, if the RF portion of input 162 is part of a separate set-top box. In various embodiments where the display 166 and speaker 168 are external components, the output signals can be submitted via a dedicated output connection, such as an HDMI port, a USB port, or a COMP output.

[0085] System 140 may include one or more sensor devices 160. Examples of sensor devices that may be used include one or more GPS sensors, gyroscopes, accelerometers, light sensors, cameras, depth cameras, microphones, and / or magnetometers. The above sensors may be used to determine information such as the user's position and orientation. If System 140 is used as a control module for an augmented reality display (e.g., a control module), the user's position and orientation may be used to determine how to render image data so that the user perceives the correct part of a virtual object or virtual scene from the correct viewpoint. In the case of a head-mounted display device, the position and orientation of the device itself may be used to determine the user's position and orientation for the purpose of rendering virtual content. In the case of other display devices, such as phones, tablets, computer monitors, and televisions, other inputs may be used to determine the user's position and orientation for the purpose of rendering content. For example, the user may use a touchscreen, keypad or keyboard, trackball, joystick, or other input to select and / or adjust a desired viewpoint and / or gaze direction. If the display device has sensors such as an accelerometer and / or gyroscope, the viewpoint and orientation used for rendering content may be selected and / or adjusted based on the movement of the display device.

[0086] The embodiments can be implemented by computer software implemented by the processor 142, by hardware, or by a combination of hardware and software. In a non-limiting example, the embodiments can be implemented by one or more integrated circuits. The memory 144 can be of any type appropriate for the technical environment and can be implemented using any suitable data storage technology, such as, for a non-limiting example, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 142 can be of any type appropriate for the technical environment and can include, for a non-limiting example, one or more of a microprocessor, a general-purpose computer, a dedicated computer, and a processor based on a multi-core architecture.

[0087] Replication interactivity Figure 2 is a system diagram illustrating an exemplary set of interfaces for an MPEG-I node hierarchy 200 supporting elements of scene interactivity according to several embodiments. According to this principle, in addition to the node tree described in relation to Figure 3, behavior metadata items (referred to here as “behaviors”) are added to the scene description. In exemplary embodiments, the time-evolving scene description is augmented by adding information that identifies the behaviors. These behaviors may relate to predefined virtual objects that allow runtime interactivity for a user-specific XR experience.

[0088] In some embodiments, the behavior described above is a time-varying behavior. In the embodiments described above, the behavior may be updated through an existing scene description update mechanism.

[0089] In exemplary embodiments, the behavior is characterized by one or more of the following characteristics: ● One or more triggers that define the conditions that must be met for activation. ● A trigger control parameter that defines the logical operations between defined triggers. ● Actions that will be implemented in response to the activation of a trigger. ●Action control parameters that define the order in which defined actions are executed. ● A priority number that allows selection of the behavior with the highest priority when multiple behaviors exist simultaneously for the same virtual object. ● An optional interrupt action that specifies how to terminate the behavior described above if the behavior is no longer defined in a newly received scene update. For example, the behavior is no longer defined if the related objects are removed, or if the behavior is no longer relevant to the current media (e.g., audio or video) sequence.

[0090] By adding the behaviors described above, we can define time-dependent user interactivity in immersive content for XR experiences.

[0091] When the second scene description is received, some of the behaviors in the first scene description are "in progress," meaning they may be triggered and actions are being performed. The second scene description may be provided as update metadata, i.e., metadata describing the differences between the first and second scene descriptions. The second scene description includes a node tree describing objects that are common or different from the objects in the first scene description. Objects in the node tree of the first scene description may no longer exist in the second description. If the second scene description lacks objects related to the actions being performed for a behavior in progress, the behavior in progress is no longer applicable. Similarly, if a behavior in progress is not defined in the second description, the behavior in progress is no longer applicable. The interrupt behavior field describes how the actions being performed should be correctly interrupted for the behavior that is being executed.

[0092] Haptic data encoding Figure 3 is a system diagram illustrating an exemplary set of interfaces for an MPEG haptic architecture according to several embodiments. Figure 3 shows example architecture 300 of a haptic codec. The new standard WG07N00624, Text of ISO / IEC DIS 23090-31 MPEG Haptics Coding Phase 1, MPEG 142 (April 2023) ("ISO / IEC23090-31") is currently under development by MPEG (Motion Picture Experts Group). Figure 3 illustrates MPEG haptic codec architecture 300. See ISO / IEC23090-31.

[0093] In the architecture 300 described above, the coded representation of haptic data can be in one of two formats: the exchange format (.hjif) 302 or the distribution format (.hmpg) 304. The exchange format is a JSON-based human-readable description of haptic data, while the distribution format is a compressed binary representation of the data. The two formats serve complementary purposes, and lossless or lossy one-to-one conversions can occur between them. An example of this conversion is shown near the center of Figure 3, where the term "interchange format" is indicated.

[0094] The binary bitstream compressed in the distribution format is structured into a series of NAL (network abstraction layer) units called "packets" to facilitate encapsulation by network protocols and file formats. This operation is illustrated by the binary compression box 306 and the binary decompression box 308, and the boxes in between.

[0095] The haptics decoder 310 takes a binary ".hmpg" or ".hjif" file as input and outputs a ".hjif" file. The haptic data contained in the resulting ".hjif" file can be rendered, for example, directly on a haptic device, or using an intermediate synthesizer 312 that generates PCM (pulse code modulation) data, as shown on the right side of Figure 3.

[0096] Figure 4 is a system diagram illustrating exemplary hierarchical data structures for two codec formats according to several embodiments. Figure 4 shows an example of haptic data hierarchy 400. The data structures of the two haptic codec formats follow the hierarchical configuration shown in Figure 4.

[0097] The top level of the structure describes the entire haptic experience defined in the file or stream. This top level includes several high-level metadata pieces 402, providing a list of avatars 404 and / or bodily representations that can be referenced to specify the desired location of haptic stimuli on the body. The haptic data itself is described through a list of perceptions 406. These perceptions 406 correspond to haptic signals associated with a particular perception modality (e.g., vibration, force, position, velocity, and temperature).

[0098] In addition to perception-specific metadata 408, perception may include a list of channels (or tracks 410) in which data is decomposed into frequency bands. Each band 412 defines a portion of the signal within a given frequency range. Bands 412 are described by a list of haptic effects 414, each containing a list of keyframes 416. The haptic signals of a channel may be reconstructed by combining data from different bands (for example, by adding high-frequency and low-frequency bands).

[0099] Figure 5 is a schematic example showing exemplary NAL unit structures in a haptic bitstream according to several embodiments. The NAL unit (NALu) structure 500 includes a header and a payload. The NALu header 502 is 32 bits, including 4 bits for the nal_unit_type field, 2 bits for the level field, 10 bits reserved for future use, and 16 bits for the payload byte length value. The nal_unit_type field may indicate a Metadata Expe type (including nb Perception, Phase 1, 2a, 2b, or Avatar), a Metadata Perception type (including nb Track, Type, Library, or Device), a Metadata Track type (including nb Band), or a DataBand type (including Header Band). The level field is the level of the bandwidth with 0 as the baseline. The metadata NALu payload 504 is n bits of metadata, where n is a variable quantity. The DataBand NALu payload 506 is a variable-length field consisting of an n*8 bit header and n sets of subfields FX1 to FXn, each with an n bit width.

[0100] Figure 6 is a schematic example showing exemplary NAL unit payload types according to several embodiments. Figure 6 shows structural examples 600 of how the structure of the NALu header 602 and NALu payload 604 can be implemented for several NAL unit types. The left side of Figure 6 shows a 16-bit NALu header 602 having 4 bits indicating the NAL type, 2 bits indicating the level, and 10 bits reserved for future use. NAL type b0000 is shown as corresponding to Metadata Haptic Experience data 606. NAL type b0001 is shown as corresponding to Metadata Haptic Perception data 608. NAL type b0010 is shown as corresponding to Metadata Haptic Track data 610. NAL type b0011 is shown as corresponding to Metadata Haptic Band data 612. NAL type b0100 is shown as corresponding to Library 614 of Effect data. NAL type b0101 is indicated as corresponding to Databand data 616. NAL type b1100 is indicated as corresponding to Cyclic Redundancy Code (CRC) 618. NAL type b1101 is indicated as corresponding to byte stuffing 620.

[0101] ISO-based media file format Figure 7 is a code listing illustrating exemplary EditListBox class structures according to several embodiments. Figure 7 shows an example of code listing 700. Within the ISO / IEC 14496 (MPEG-4) standard, there are several parts that define file formats for storing time-based media. According to ISO / IEC 14496-12, Coding of Audio-Visual Objects, Part 12: ISO Base Media File Format, 2020 ("ISO / IEC 14496-12"), all these parts are based on and derived from ISOBMFF (ISO Base Media File Format), which is a structured and media-independent definition. ISOBMFF mainly contains structural information and media data information for the timed presentation of media data such as audio and video. Furthermore, it has support for untimed data, such as different levels of metadata within the file structure. The logical structure of a file is a movie containing a set of time-parallel tracks. The time structure of a file is such that the tracks contain a sequence of samples in time, and those sequences are mapped onto the timeline of the entire movie. ISOBMFF is based on the concept of a box structure file. A box structure file has a set of boxes (sometimes called atoms) that have a size and a type. The type is a 32-bit value, usually consisting of four printable characters, also known as a four-character code (4CC). Untimed data is attached to one of the timed data streams, called a metadata box, file-level, movie box, or track within a movie.

[0102] Among the top-level boxes within an ISOBMFF container, the MovieBox ("moov") contains metadata for the continuous media stream present in the file. This metadata is signaled within the box hierarchy of the MovieBox, for example, within the TrackBox ("trak"). A track represents a continuous media stream present in the file. The media stream itself is a sequence of samples, such as audio and video access units of an elementary media stream, and is enclosed within a MediaDataBox ("mdat") at the top level of the container. The metadata for each track includes a list of sample description entries, each providing the encoding or encapsulation format used in the track and initialization data for processing that format. Each sample is associated with one of the track's sample description entries. ISO / IEC 14496-12 provides a tool for defining an explicit timeline map for each track. This is known as an edit list and is indicated using an EditListBox with the syntax shown in Figure 7. Each entry defines a portion of the track timeline. Map a portion of the composition timeline, or indicate "empty" time (a portion of the presentation timeline that does not map media, "empty" editing).

[0103] Figure 7 shows an example of the class structure of the EditListBox extension of the FullBox class described above. In the example in Figure 7, a sequence of samples numbered from 1 onwards relative to the value of entry_count forms the media stream. Figure 7 shows support for "version 0" and "version 1". Version 0 uses 32-bit integers for the sample duration (variable edit_duration) and the time in the media stream (variable media_time), while version 1 uses 64-bit integer fields. Each sample also has media_rate_integer and media_rate_fraction values ​​corresponding to the sampling rate of the sample.

[0104] Haptic Data Transmission MPEG is currently working on a new standard, WG03N00686, Text of ISO / IEC CD 23090-32 Carriage of Haptics Data, MPEG142, April 2023 (ISO / IEC 23090-32), which defines how haptic bitstreams generated by the ISO / IEC 23090-31 codec are encapsulated in the ISOBMFF media container. The current version of ISO / IEC CD23090-32 describes how haptic data is transmitted on a single track within a file.

[0105] Application 638 describes a more flexible and scalable design for transmitting haptic data using a multi-track approach. In multi-track mode, a haptic experience takes place across multiple ISOBMFF tracks, with one track serving as the main track of the experience, conveying general information applicable to the entire experience. The main track may be associated with one or more haptic tracks, through track references in the ISOBMFF file, carrying haptic band data units for one or more haptic channels. Samples in this track may transmit data for only one band of a perceptual channel, or data for all bands of a perceptual channel. If all bands are included in the sample of the haptic track, each sample may consist of subsamples, each subsample containing data for any of the bands.

[0106] The bitstream format developed under ISO / IEC 23090-31 allows for describing haptic experiences in a compact representation that can be easily consumed by haptic decoders; however, currently there is no clearly defined, standardized method for streaming such encoded haptic content. A method for storing ISO / IEC 23090-31 encoded haptic bitstreams in an ISOBMFF container by demultiplexing data belonging to different haptic channels into separate tracks within a file was presented in m61136, "Haptics" On Carriage of Haptics Data in ISOBMFF, MPEG 140, October 2022. While this design allows for scalable access to different haptic stream components in local playback scenarios, such as when all data is stored and accessed locally, the above structure may not be directly usable for remote access and adaptive streaming of haptic content stored on remote servers over a network.

[0107] The application demonstrates a method for supporting adaptive streaming of haptic bitstreams generated by ISO / IEC 23090-31 and packaged in an ISOBMFF container as multiple tracks. This application describes, in several embodiments, methods and systems that enable flexible and scalable streaming of haptic media that can be coded using the ISO / IEC 23090-31 codec developed by ISO / IEC SC29 / WG03 (MPEG Systems).

[0108] With respect to some embodiments, the ideas presented in this application may be applied to immersive media coding, encoding, storage, and streaming of encoded haptic media content, as well as decoding of haptic media data in a device or any service that provides an immersive media experience.

[0109] Dynamic Streaming over HTTP (DASH) MPEG-DASH (MPEG Dynamic Adaptive Streaming over HTTP) is a universal distribution format that dynamically adapts to changing network conditions to provide end users with the best possible video experience. Dynamic HTTP streaming requires that a variety of bitrate options for multimedia content be available on the server. Furthermore, multimedia content may consist of multiple media components (audio, video, text, etc.), each with different characteristics. In MPEG-DASH, these characteristics are described by MPD (Media Presentation Description).

[0110] Figure 8 is a schematic example illustrating an exemplary MPD hierarchical data model according to several embodiments. The exemplary hierarchical data model 800 includes a series of hierarchical boxes. MPD 802 describes a sequence of periods 804 in which a consistent set of encoded versions of media content components remains unchanged during the period. Each period has a start time and duration and consists of one or more AdaptationSets.

[0111] An AdaptationSet 806 represents a set of encoded versions of one or more media content components that share identical properties, such as language, media type, image aspect ratio, role, accessibility, and rating properties. For example, an AdaptationSet might contain video components of the same multimedia content at different bitrates. Another AdaptationSet might contain audio components of the same multimedia content at different bitrates (e.g., low-quality stereo and high-quality surround). Each AdaptationSet typically contains multiple Representations.

[0112] A Representation808 describes a deliverable encoded version of one or more media components that differs from other representations by bitrate, resolution, number of channels, or other characteristics. Each Representation consists of one or more segments. For example, attributes of a Representation element, such as id, @bandwidth, @qualityRanking, and @dependencyId, are used to specify properties of the associated Representation. A Representation may also contain Sub-Representations, which are part of the Representation, to describe and extract partial information from the Representation. A Sub-Representation may provide access to a lower-quality version of the Representation it contains.

[0113] Segment 810 is the largest unit of data that can be retrieved in a single HTTP request. Each segment has a URL, an addressable location on the server, and can be downloaded using HTTP GET or HTTP GET with a specified byte range.

[0114] To use the data model, the DASH client parses the MPD XML document and selects a collection of AdaptationSets appropriate for its environment based on the information provided by each element of the AdaptationSet. Within each AdaptationSet, the client selects one Representation, typically based on the value of the @bandwidth attribute, but also considering the client's decoding and rendering capabilities. The client downloads the initialization segment of the selected Representation and accesses the content by requesting the entire segment or a byte range of the segment. Once the presentation begins, the client continues to consume media content by continuously requesting media segments or portions of media segments and playing the content according to the media presentation timeline. The client may switch Representations to account for updates from its environment. The client must continuously play content across periods. After consuming the media contained in the segment toward the end of the media announced in the Representation, the client may end the media presentation and start a new Period or refetch the MPD.

[0115] DASH descriptor MPEG-DASH introduces the concept of descriptors to provide application-specific information about media content. Descriptors are all structured in the same way, including the `@schemeIdUri` attribute, which provides a URI to identify the scheme, and optional attributes `@value` and `@id`. The semantics of the element are specific to the scheme being used. The URI identifying the scheme is either a URN or a URL. MPD does not provide specific information on how to use these elements. Instantiating the description element with appropriate scheme information is up to the application adopting the DASH format. A DASH application using one of the elements described above must first define the Scheme Identifier in the form of a URI, and then define the value space of the element where the Scheme Identifier is used. Extended elements and attributes may be defined in separate namespaces when structured data is required. Descriptors can appear at multiple levels within MPD. ● The existence of an element at the MPD level means that the element is a child of an MPD element. ● The existence of an element at the adaptation set level indicates that the element is a child element of an adaptation set element. ● The existence of an element at the representation level refers to the fact that the element is a child element of a Representation element.

[0116] Pre-Selection In MPEG-DASH, a bundle is a set of media components that may be jointly consumed by a single decoder instance. Each bundle contains decoder-specific information and a main media component that bootstraps the decoder. PreSelection defines a subset of media components within a bundle that are expected to be jointly consumed.

[0117] An AdaptationSet containing the main media component is called the main AdaptationSet. The main media component is always included in the PreSelection associated with the bundle. In addition, each bundle may contain one or more subAdaptationSets. Some AdaptationSets may only be processed in combination with the main AdaptationSet.

[0118] PreSelection can be defined through the PreSelection element defined in Table 1. PreSelection selection is based on the attributes and elements contained within the PreSelection element. In Table 1, attributes designated as required are listed with a Use value of "M", and attributes designated as optional are listed with a Use value of "O". An "OD" entry means that the attribute is optional and a default value is listed. For elements, the Use value lists the minimum and maximum occurrences. An "N" entry means that the number of occurrences is unlimited. Elements are shown first in bold, while attributes are preceded by non-bold text and begin with the "@" symbol.

[0119] [Table 1]

[0120] Haptic media adaptation set Figure 9 is a system diagram illustrating exemplary DASH configurations for grouping adaptation sets according to several embodiments. Figure 9 shows an example of a DASH configuration 900 for grouping adaptation sets 904, 906, 908, and 910 belonging to the same haptics experience 902 within an MPEG-DASH MPD file.

[0121] In the DASH Media Presentation Descriptor (MPD), to signal the presence of multi-track haptics media, each track of the media, including the main track, is represented by an AdaptationSet element in the MPD. The adaptation set 904 for the main (haptics experience) track is called the Haptics Experience Adaptation Set, and the adaptation sets 906, 908, and 910 for the related haptics tracks are called the Haptics Adaptation Set.

[0122] 4CC is a four-character code used in ISOBMFF to identify the type of sample entry in a track, based on the codec type (for example, "hev1" is the 4CC for the HEVC codec with a specific configuration). Tracks are represented by an Adaptation Set in the DASH MPD. Typically, the @codecs attribute is set to the corresponding 4CC for the relevant track. In some embodiments, the same 4CC may be used for both haptics experience ASs and haptics data ASs, as they use the same haptics codec.

[0123] In some embodiments, perception is part of the same haptic experience. A haptic experience includes many perceptions, each perception having several channels, and each channel having several bandwidths. A haptic data adaptation set represents a track that carries the bandwidth data of one or more channels of a particular perception in a haptic experience.

[0124] If the Haptics Experience Adaptation Set has the @codecs attribute set to "mih1" and the AdaptationSet element does not have the @codecs attribute, then each of the Haptics Adaptation Sets (or the Representations of these Adaptation Sets) will have the @codecs attribute set to "mihb". The @mimeType attribute of all Haptics Experience adaptation sets will be set to "haptic / mp4", which is the MIME type registered for haptic media.

[0125] A Haptics Experience Adaptation Set includes a single adaptation set-level Initialization Segment. The Initialization Segment contains all the MIHS (MPEG-I Haptic Stream) units (packets) necessary to initialize the haptics decoder. The media segment for the Haptics Adaptation Set's Representation contains one or more track fragments of the corresponding haptic track at the file format level. By concatenating the Initialization Segment with media segments from one or more Haptics Adaptation Sets, the resulting file contains a bitstream that can be decoded by the haptics decoder.

[0126] The Representation in the Haptics Adaptation Set of Haptics Experience Preselection has an @dependencyId attribute that sets the ID of the Representation in the corresponding Haptics Experience Adaptation Set.

[0127] If a Haptics Adaptation Set contains multiple Representations, the @bitstreamSwitching attribute, present in the AdaptationSet element of the Haptics Adaptation Set and set to "true," indicates to the player that seamless switching between Representations within the Adaptation Set is supported. Furthermore, the duration of the media segments for each Representation must be identical.

[0128] For example, if the segments are not identical, when the player switches to a different representation, the player must determine where in the new media segment obtained from the other representation it should continue playback from. Furthermore, the player must calculate which segments it needs from the last timestamp of the last segment played from the first representation. In other words, in this case, the switch is not seamless; the segment indices are not aligned in time. Therefore, according to some embodiments, the duration of the media segments in each representation must be identical.

[0129] Haptics Experience Preselection Haptics Experience Preselection may be signaled within an MPD using a PreSelection element within a Period element or a PreSelection descriptor at the Adaptation Set level. The Haptics Experience PreSelection element is signaled with the @PreSelectionComponents attribute, as defined in ISO / IEC 23009-1, and the assigned value is a list of IDs, which include the IDs of the Haptics Experience Adaptation Sets followed by the ID of the associated Haptics Adaptation Set. The @codecs attribute of the PreSelection is set to "mih1" to indicate that the media represented by the PreSelection is coded haptic media.

[0130] In some embodiments, the adaptation set may include information that identifies one or more fragments of the corresponding haptic track.

[0131] Haptics Experience Descriptor Table 2 lists the elements and attributes of haptic descriptors, which may be contained in separate files. In Table 2, attributes designated as required are listed with a Use value of "M", and attributes designated as optional are listed with a Use value of "O". For elements, the Use value lists the minimum and maximum occurrences. An entry with "N" means that the number of occurrences is unlimited. Elements are shown first in bold, while attributes precede them in plain text and begin with the "@" symbol.

[0132] To represent different perceptions within a haptics experience, a HapticsExperience descriptor is created for the Haptics Experience Adaptation Set. In some embodiments, the HapticsExperience descriptor is an EssentialProperty descriptor with the @schemeIdUri attribute set to a unique URI (Uniform Resource Identifier) ​​(e.g., "urn:mpeg:mpegI:haptics:2023:perception"). The @value attribute of the HapticsExperience descriptor is absent. The HapticsExperience descriptor contains elements and attributes that describe the perception associated with the haptics experience. The Haptics descriptor contains at least one or more hapticsPerception elements, each containing an @id attribute set to a unique identifier for the perception in the haptics bitstream and an @type attribute indicating its modality.

[0133] [Table 2]

[0134] Figure 10 is a code listing illustrating exemplary XML schemas according to several embodiments. Figure 10 is an example of code listing 1000, which shows examples of data types for various elements and attributes of the XML schema. In the example shown in Figure 10, line 6 of the code listing sets the element name to "hapticsPerception" and the data type to "haptics:HapticsPerceptionType". Line 8 of the code listing sets the name, type, and purpose for the @id attribute. Line 9 of the code listing sets the name, type, and purpose for the @type attribute.

[0135] In some embodiments, the HapticsExperiece descriptor is a SupplementalProperty descriptor whose @schemeIdUri attribute is set to a unique URI (e.g., "urn:MPEG:MPEGI:haptics:2023:perception").

[0136] Haptics channel descriptor Table 3 lists the elements and attributes of haptic descriptors, which may be contained in separate files. In 32, attributes designated as required are listed with a Use value of "M", and attributes designated as optional are listed with a Use value of "O". For elements, the Use value lists the minimum and maximum occurrences. An entry with "N" means that the number of occurrences is unlimited. Elements are shown first in bold, while attributes precede them in plain text and begin with the "@" symbol.

[0137] Haptics descriptors are used to identify haptic channels present in a Haptics Adaptation Set. In some embodiments, a Haptics descriptor is an EssentialProperty descriptor with @schemeIdUri set to a unique URI (e.g., "urn:MPEG:MPEGI:haptics:2023:channel"). At the adaptation set level, a Haptics descriptor signals each haptic channel present in the Representation of the Haptics Adaptation Set. Again, the @value attribute of the Haptics descriptor is not present. The Haptics descriptor is associated with a Haptics Data Adaptation Set and provides information about the channels and bandwidth carried in that Adaptation Set. The HapticsExperience descriptor provides a higher level of metadata about the haptics experience itself. In this case, it is a list of perceptions available in the experience. The hapticChannel element of the Haptics descriptor points to one of these perceptions.

[0138] [Table 3]

[0139] Figure 11 is a code listing illustrating exemplary XML schemas according to several embodiments. Figure 11 shows an example of code listing 1100. The data types of various elements and attributes are defined in the example XML schema shown in Figure 10. In the example shown in Figure 11, line 6 of the code listing sets the element name to "hapticsChannel" and the data type to "haptics:HapticsChannelType". Line 8 of the code listing sets the name, type, and purpose of the @id attribute. Line 9 of the code listing sets the name, type, and purpose of the @perceptionId attribute. Line 10 of the code listing sets the name, type, and purpose of the @band_ids attribute.

[0140] Figure 12 is a flowchart illustrating exemplary processes for encoding haptic data according to several embodiments. In some embodiments, exemplary process 1200 may include encoding information describing a haptic experience 1202. In some embodiments of exemplary process 1200, the information describing the haptic experience includes one or more adaptation sets 1204. In some embodiments of exemplary process 1200, each of the one or more adaptation sets includes one or more representations corresponding to a haptic media track 1206.

[0141] Figure 13 is a flowchart illustrating exemplary processes for decoding haptic data according to several embodiments. In some embodiments, exemplary process 1300 may include decoding information describing a haptic experience 1302. In some embodiments of exemplary process 1300, the information describing a haptic experience includes one or more adaptation sets 1304. In some embodiments of exemplary process 1300, each of the one or more adaptation sets includes one or more representations corresponding to a haptic media track 1306.

[0142] Methods and systems according to some embodiments are generally discussed in the context of XR (extended reality), but some embodiments may be applicable to any XR context, such as in the context of VR (virtual reality) / MR (mixed reality) / AR (augmented reality). Also, the term “head-mounted display (HMD)” is used herein according to some embodiments, but for some embodiments it may be applied to wearable devices (which may or may not be mounted on the head) capable of XR, VR, AR, and / or MR.

[0143] An exemplary method, according to several embodiments, includes encoding information describing a haptic experience, the information describing the haptic experience may include one or more sets of adaptations, each of which may include one or more representations corresponding to a haptic media track.

[0144] With respect to some embodiments of the exemplary method, each of one or more expressions corresponds to the same period of time.

[0145] In some embodiments of the exemplary method, information encoding involves encoding the information in a container file.

[0146] In some embodiments of the exemplary method, one or more adaptation sets may include a main haptics experience and a second haptics experience, the second haptics experience corresponding to a first perception modality and a first channel.

[0147] In some embodiments of the exemplary method, the adaptation set corresponding to the main haptics experience may include initialization data corresponding to the haptics decoder.

[0148] With respect to some embodiments of the exemplary method, the adaptation set corresponding to the second haptic experience may include one or more fragments of the corresponding haptic track.

[0149] In some embodiments of the exemplary method, the adaptation set corresponding to the second haptic experience may include information identifying one or more fragments of the corresponding haptic track.

[0150] In some embodiments of the exemplary method, the first channel corresponds to a first frequency band.

[0151] With respect to some exemplary embodiments of the method, the information describing the haptic experience may further include information identifying one or more sets of adaptations.

[0152] With respect to some embodiments of the exemplary method, the information describing the haptic experience may further include information describing at least one available avatar for the haptic experience, and configuration information for at least one perception in the haptic experience, the configuration information may include information describing one or more parallel haptic experience tracks.

[0153] With respect to some embodiments of the exemplary method, the information may further include information describing one or more haptic experience tracks, the information describing the haptic experience tracks may further include information describing at least one available avatar for the haptic experience, and configuration information for at least one perception in the haptic experience.

[0154] An exemplary method / apparatus, according to several embodiments, includes a processor and a non-temporary computer-readable medium that stores instructions operable to cause the apparatus to encode information describing a haptic experience, the information describing a haptic experience may include one or more sets of adaptations, each of which includes one or more representations corresponding to a haptic media track.

[0155] An exemplary additional method, according to several embodiments, includes decoding information describing a haptic experience, which may include one or more adaptation sets, each of which includes one or more representations corresponding to a haptic media track.

[0156] With respect to some embodiments of the additional exemplary methods, each of one or more expressions corresponds to the same period of time.

[0157] Regarding some embodiments of additional exemplary methods, information encoding involves encoding information in a container file.

[0158] With respect to some embodiments of additional exemplary methods, one or more adaptation sets may include a main haptic experience and a second haptic experience, the second haptic experience corresponding to a first perception modality and a first channel.

[0159] In some embodiments of additional exemplary methods, the adaptation set corresponding to the main haptics experience may include initialization data corresponding to the haptics decoder.

[0160] With respect to some embodiments of additional exemplary methods, the adaptation set corresponding to the second haptic experience may include one or more fragments of the corresponding haptic track.

[0161] With respect to some embodiments of additional exemplary methods, the adaptation set corresponding to a second haptic experience may include information identifying one or more fragments of the corresponding haptic track.

[0162] Some embodiments of the additional exemplary methods may further include generating a bitstream by concatenating initialization data with information that identifies one or more fragments from one or more adaptation sets.

[0163] Some embodiments of the additional exemplary methods may further include rendering a bitstream in a haptic experience environment.

[0164] In some embodiments of the additional exemplary method, the first channel corresponds to a first frequency band.

[0165] With respect to some embodiments of additional exemplary methods, the information describing the haptic experience may further include information identifying one or more sets of adaptations.

[0166] With respect to some embodiments of additional exemplary methods, the information describing the haptic experience may further include information describing at least one available avatar for the haptic experience, and configuration information for at least one perception in the haptic experience, the configuration information may include information describing one or more parallel haptic experience tracks.

[0167] With respect to some additional exemplary embodiments of the method, the information may further include information describing one or more haptic experience tracks, the information describing the haptic experience tracks may further include information describing at least one available avatar for the haptic experience, and configuration information for at least one perception in the haptic experience.

[0168] With respect to some embodiments of the exemplary method, the information describing the haptic experience, and / or the encoded information describing the haptic experience, is encoded according to ISOBMFF (ISO Base Media File Format).

[0169] In some embodiments of the exemplary method, the information describing the haptic experience, and / or the encoded information describing the haptic experience, is contained in a media file, such as an MPD (media presentation descriptor) file for MPEG-DASH (MPEG Dynamic Adaptive Streaming Over HTTP), and is streamed.

[0170] With respect to some embodiments of the exemplary method, the information describing the haptic experience, and / or the encoded information describing the haptic experience, is one or more parts of, and / or conforms to, one or more of the MPEG standards ISO / IEC 23090-32 and ISO / IEC 23090-31.

[0171] An exemplary method / apparatus, according to several embodiments, includes a processor and a non-temporary computer-readable medium that stores instructions, when executed by the processor, causing the apparatus to decode information describing a haptic experience, the information describing a haptic experience may include one or more sets of adaptations, each of which includes one or more representations corresponding to a haptic media track.

[0172] Exemplary devices according to several embodiments may include at least one processor configured to perform one of the methods listed above.

[0173] Exemplary devices according to some embodiments may include a computer-readable medium that stores instructions for causing one or more processors to perform any one of the methods listed above.

[0174] Exemplary devices according to some embodiments may include at least one processor and at least one non-temporary computer-readable medium for storing instructions for causing the at least one processor to perform one of the methods listed above.

[0175] Exemplary devices according to several embodiments may include a computer-readable medium for storing scene description files of encoded information generated according to any one of the methods listed above.

[0176] An exemplary signal according to several embodiments may include a scene description file generated according to one of the methods listed above.

[0177] This disclosure describes various aspects, including tools, features, embodiments, models, and approaches. Many of these aspects are described in a specific manner and, at least to illustrate their individual characteristics, often in a way that may give the impression of limitation. However, this is for the purpose of clarity in the description and does not limit the disclosure or scope of these aspects. Indeed, all of the different aspects can be combined and interchangeable to provide further aspects. Furthermore, these aspects can also be combined and interchangeable with aspects described in earlier applications.

[0178] The embodiments described and envisioned in this disclosure can be implemented in many different ways. While some embodiments are specifically illustrated, other embodiments are envisioned, and discussion of specific embodiments does not limit the scope of implementation. Generally, at least one of the embodiments relates to video encoding and decoding, and generally, at least one other embodiment relates to transmitting a generated or encoded bitstream. The embodiments described herein and other embodiments can be implemented as computer-readable recording media storing methods, apparatus, instructions for encoding or decoding video data according to any of the described methods, and / or computer-readable recording media storing a bitstream generated according to any of the described methods.

[0179] In this disclosure, the terms “reconstructed” and “decoded” may be used interchangeably, the terms “pixel” and “sample” may be used interchangeably, and the terms “image,” “picture,” and “frame” may be used interchangeably. Typically, though not always, the term “reconstructed” is used on the encoder side, while “decoded” is used on the decoder side.

[0180] The terms HDR (High Dynamic Range) and SDR (Standard Dynamic Range) often refer to specific values ​​of dynamic range for those skilled in the art. However, additional embodiments are also intended in which reference to HDR is understood to mean "higher dynamic range" and reference to SDR is understood to mean "lower dynamic range." These additional embodiments are not limited by any specific values ​​of dynamic range that may often be associated with the terms "High Dynamic Range" and "Standard Dynamic Range."

[0181] Various methods are described herein, each of which includes one or more steps or actions to achieve the described method. Unless a particular order of steps or actions is required for the inherent operation of the method, the order and / or use of certain steps and / or actions may be modified or combined. In addition, terms such as “first,” “second,” etc., may be used in various embodiments that modify elements, components, steps, actions, etc., such as “first decode” and “second decode.” The use of such terms does not imply ordering of the modified operations unless specifically required. Thus, in the examples given, the first decode does not need to be performed before the second decode, and may occur, for example, before the second decode, during the second decode, or in a time period that overlaps with the second decode.

[0182] For example, various numerical values ​​may be used in this disclosure. Certain values ​​are for illustrative purposes only, and the embodiments described are not limited to the specific values ​​mentioned above.

[0183] The embodiments described herein may be implemented by computer software implemented by a processor, by hardware, or by a combination of hardware and software. In a non-limiting example, embodiments may be implemented by one or more integrated circuits. The processor may be of any type suitable for the technical environment and may include, in a non-limiting example, one or more of microprocessors, general-purpose computers, dedicated computers, and processors based on multicore architectures.

[0184] Various implementations include decoding. “Decoding” as used in this disclosure can encompass all or part of the processing performed on the received encoded sequence to produce a final output suitable for display. In various embodiments, the processing includes one or more of the processing typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, the processing also includes, or alternatively, processing performed by the decoders of the various implementations described in this disclosure, such as extracting pictures from a tiled (packed) picture, determining the upsampling filter to use and then upsampling the picture, and flipping the picture back to its intended orientation.

[0185] As further examples, in one embodiment, "decoding" refers only to "entropy decoding," in another embodiment, "decoding" refers only to differential decoding, and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to refer more broadly in general, the decoding process will become clear based on the context of the particular explanation.

[0186] Various implementations include encoding. In a manner similar to the above description of “decoding,” “encoding” as used in this disclosure can encompass, for example, all or part of the processing performed on the input video sequence to generate the bitstream being encoded. In various embodiments, the processing includes one or more of the processing typically performed by an encoder, such as splitting, differential encoding, transformation, quantization, and entropy encoding. In various embodiments, the processing may further or alternatively include processing performed by the encoders of the various implementations described in this disclosure.

[0187] As further examples, in one embodiment, "encoding" refers only to "entropy decoding," in another embodiment, "encoding" refers only to differential decoding, and in yet another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to refer more broadly in general, the encoding process will become clear based on the context of the particular description.

[0188] Various embodiments cite rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is typically considered, often given constraints on computational complexity. Typically, rate-distortion optimization is formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, an approach may be based on extensive testing of all encoding options, including all considered mode or encoding parameter values, along with a full evaluation of the encoding cost and associated distortion of the reconstructed signal after encoding and decoding. Furthermore, faster approaches may be used, in particular, with the calculation of approximated distortion based on a predicted or predicted residual signal rather than one of the reconstructed signals, in order to eliminate the complexity of encoding. A mixture of the two approaches described above can also be used, for example, by using approximated distortion for only some of the possible encoding options and full distortion for the other encoding options. Other approaches evaluate only a subset of the possible encoding options. More generally, many approaches employ one of several optimization techniques, but optimization does not necessarily provide a complete evaluation of both the coding cost and the associated distortions.

[0189] When a diagram is given as a flowchart, it should be understood that it also provides a block diagram of the corresponding device. Similarly, when a diagram is given as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0190] The implementations and embodiments described herein can be implemented, for example, in methods or processes, apparatus, software programs, data streams, or signals. Even when described only in the context of a single form of implementation (e.g., only as a method), the implementation of the described features can also be implemented in other forms (e.g., apparatus or programs). Apparatus can be implemented, for example, in appropriate hardware, software, and firmware. Methods can be implemented, for example, in processors, which generally refer to processing devices, including computers, microprocessors, integrated circuits, or programmable logic devices. Furthermore, processors also include communication devices, such as computers, cell phones, portable / personal digital assistants (PDAs), and other devices that facilitate the communication of information between end users.

[0191] References to “one embodiment,” “embodiment,” “one implementation,” or “implementation” mean that, like other variations, certain features, structures, characteristics, etc., described in relation to the embodiment are included in at least one embodiment. Thus, the appearances of the phrases “in one embodiment,” “in one embodiment,” or “in one implementation,” or “in implementation,” which appear in various places throughout this disclosure, do not necessarily refer to all of the same embodiments, like any other variation.

[0192] In addition, this disclosure may refer to “determining” various parts of information. Determining information may include, for example, one or more of the following: estimating information, calculating information, predicting information, or retrieving information from memory.

[0193] Furthermore, this disclosure may refer to “accessing” various parts of the information. Accessing the information may include, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0194] In addition, this disclosure may refer to various parts of information as “receiving.” Receiving is intended to be a broad term, as with respect to “accessing.” Receiving information can include, for example, accessing information or retrieving information (for example, from memory). Furthermore, “receiving” usually includes, in some way or otherwise, between actions such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0195] It should be understood that the use of any of the following " / ", "and / or", and "at least one of" is intended to include, for example, "A / B", "A and / or B", and "at least one of A and B", which include selecting only the first enumerated option (A), or only the second enumerated option (B), or both options (A and B). As further examples, in the case of "A, B, and / or C" and "at least one of A, B, and C", the above phrases are intended to include selecting only the first enumerated option (A), or only the second enumerated option (B), or only the third enumerated option (C), or only the first and second enumerated options (A and B), or only the first and third enumerated options (A and C), or the second and third enumerated options (B and C), or all three options (A, B, and C). What I've just described can be extended to many of the same items that are listed.

[0196] Furthermore, the word “signal,” as used herein, refers, among other things, to indicating something to a corresponding decoder. For example, in one embodiment, an encoder signals one particular parameter of several parameters relating to region-based filter parameter selection for de-artifact filtering. Thus, in this embodiment, the same parameter is used on both the encoder and decoder sides. Therefore, for example, the encoder can transmit a particular parameter to the decoder (explicit signaling), and the decoder can use the same particular parameter. Conversely, if the decoder already has a particular parameter, the signaling can be used without transmission (implicit signaling), simply to allow the decoder to know and select the particular parameter. By avoiding transmission in any real-world function, bit savings are achieved in various embodiments. It is understood that signaling can be performed in various ways. For example, one or more syntax elements, flags, etc., are used in various embodiments to signal information to the corresponding decoder. The above concerns the verb form of the word "signal," but the word "signal" can also be used as a noun in this specification.

[0197] An implementation can generate various signals formatted to carry information that can be stored or transmitted. For example, the information can include instructions for doing something or data generated by one of the implementations described. For example, a signal can be formatted to carry a bitstream of the embodiment described. For example, the above signal can be formatted as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. For example, formatting can include encoding a data stream and modulating a carrier wave with the encoded data stream. For example, the information carried by the signal can be analog or digital information. The signal can be transmitted over various separate wired or wireless links, as is known. The signal can be stored in a processor-readable medium.

[0198] We will describe many embodiments. The features of the embodiments described above can be provided individually or in any combination across various claim categories and types. Furthermore, embodiments can include one or more of the following features, devices, or aspects, individually or in any combination, across various claim categories and types. ● A bitstream or signal containing one or more of the described syntax elements, or variations thereof. ● A bitstream or signal containing a syntax that conveys information generated according to any of the embodiments described. ● Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal that contains one or more of the described syntax elements or variations thereof. ● Creating and / or transmitting and / or receiving and / or decoding according to any of the embodiments described. ● A method, process, apparatus, medium for storing instructions, medium for storing data, or signal according to any of the embodiments described.

[0199] It should be noted that the various hardware elements of one or more embodiments described herein are referred to as “modules” that perform (i.e., do, execute, and similar) the various functions described herein in relation to each module. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices) that would be considered appropriate by those skilled in the art relating to the given implementation. Furthermore, it should be noted that each described module may include executable instructions for performing one or more functions described as being performed by each module, and such instructions may take the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or similar forms, and may be stored in any suitable non-temporary computer-readable medium or media, such as commonly referred to as RAM, ROM, etc.

[0200] While features and elements are described above in specific combinations, those skilled in the art will understand that each feature or element can be used alone or in any combination with other features and elements. In addition, the methods described herein may be implemented in computer programs, software, or firmware embedded on computer-readable media for execution by a computer or processor. Examples of computer-readable recording media include, but are not limited to, ROM (described), RAM (random access memory), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media such as CD-ROM discs and DVDs (digital versatile disks). Processors associated with software may be used to implement radio frequency transceivers for use in UEs, WTRUs, terminals, base stations, RNCs, or any host computer.

Claims

1. This involves encoding information that describes a haptic experience. The information describing the haptic experience includes one or more adaptation sets, Each of the one or more adaptation sets includes one or more representations corresponding to a haptic media track. thing A method characterized by comprising:

2. The method according to claim 1, characterized in that each of the one or more expressions corresponds to the same period of time.

3. The method according to 1 or 2, characterized in that the encoding of the information is the encoding of the information in a container file.

4. The one or more adaptation sets include a main haptic experience and a second haptic experience. The second haptic experience corresponds to the first perception modality and the first channel. The method according to any one of claims 1 to 3, characterized by the features described herein.

5. The method according to 4, characterized in that the adaptation set corresponding to the main haptic experience includes initialization data corresponding to a haptic decoder.

6. The method according to 4 or 5, characterized in that the adaptation set corresponding to the second haptic experience includes one or more fragments of the corresponding haptic track.

7. The method according to 6, characterized in that the adaptation set corresponding to the second haptic experience includes information identifying one or more fragments of the corresponding haptic track.

8. The method according to any one of claims 4 to 7, characterized in that the first channel corresponds to a first frequency band.

9. The method according to any one of claims 1 to 8, wherein the information describing the haptic experience further includes information identifying one or more adaptation sets.

10. The information describing the aforementioned haptic experience is: Information describing at least one available avatar for the aforementioned haptic experience, Configuration information relating to at least one perception in the haptic experience, wherein the configuration information includes information describing one or more parallel haptic experience tracks. The method according to any one of claims 1 to 9, further comprising:

11. The aforementioned information further includes information describing one or more haptic experience tracks, The information describing the haptic experience track is, Information describing at least one available avatar for the haptic experience, Configuration information relating to at least one perception in the aforementioned haptic experience and Includes The method according to any one of claims 1 to 9, characterized by the following:

12. Processor and When executed by the aforementioned processor, Encode the information that describes the haptic experience, The information describing the haptic experience includes one or more adaptation sets, Each of the one or more adaptation sets includes one or more representations corresponding to a haptics media track. A non-temporary computer-readable medium storing operable commands that cause the device to do the above, An apparatus characterized by being equipped with

13. This involves decoding information that describes a haptic experience. The information describing the haptic experience includes one or more adaptation sets, Each of the one or more adaptation sets includes one or more representations corresponding to a haptic media track. thing A method characterized by comprising:

14. The method according to 13, characterized in that each of the one or more expressions corresponds to the same period of time.

15. The method according to 13 or 14, characterized in that the encoding of the information is to encode the information in a container file.

16. The one or more adaptation sets include a main haptic experience and a second haptic experience. The second haptic experience corresponds to the first perception modality and the first channel. The method according to any one of claims 13 to 15, characterized by...

17. The method according to 16, characterized in that the adaptation set corresponding to the main haptic experience includes initialization data corresponding to a haptic decoder.

18. The method according to 17, characterized in that the adaptation set corresponding to the second haptic experience includes one or more fragments of the corresponding haptic track.

19. The method according to 18, characterized in that the adaptation set corresponding to the second haptic experience includes information identifying one or more fragments of the corresponding haptic track.

20. To generate a bitstream by concatenating the information identifying one or more fragments from one or more adaptation sets with the initialization data. The method according to 19, further comprising:

21. The method according to 20, further comprising rendering the bitstream in a haptic experience environment.

22. The method according to any one of claims 16 to 21, characterized in that the first channel corresponds to a first frequency band.

23. The method according to any one of claims 13 to 22, wherein the information describing the haptic experience further includes information identifying one or more adaptation sets.

24. The information describing the haptic experience is, Information describing at least one available avatar for the aforementioned haptic experience, Configuration information relating to at least one perception in the haptic experience, wherein the configuration information includes information describing one or more parallel haptic experience tracks. Includes The method according to any one of claims 13 to 23, characterized by...

25. The aforementioned information further includes information describing one or more haptic experience tracks, The information describing the haptic experience track is, Information describing at least one available avatar for the haptic experience, Configuration information relating to at least one perception in the aforementioned haptic experience and Includes The method according to any one of claims 13 to 23, characterized by...

26. The method according to any one of claims 1 to 25, characterized in that the information describing the haptic experience and / or the encoded information describing the haptic experience is encoded in accordance with the ISO-based media file format (ISOBMFF).

27. The method according to any one of claims 1 to 25, characterized in that the information describing the haptic experience and / or the encoded information describing the haptic experience are contained in a media file such as an MPEG-DASH (MPEG Dynamic Adaptive Streaming over HTTP) Media Presentation Descriptor (MPD) file and streamed.

28. The method according to any one of claims 1 to 25, characterized in that the information describing the haptic experience and / or the encoded information describing the haptic experience is one or more parts of and / or conforms to one or more of the MPEG standards ISO / IEC 23090-32 and ISO / IEC 23090-31.

29. Processor and When executed by the aforementioned processor, Decode the information describing the haptic experience, The information describing the haptic experience includes one or more adaptation sets, Each of the one or more adaptation sets includes one or more representations corresponding to a haptics media track. A non-temporary computer-readable medium storing operable commands that cause the device to do the above, An apparatus characterized by being equipped with

30. An apparatus comprising at least one processor configured to perform the method described in any one of claims 1 to 29.

31. An apparatus characterized by comprising a computer-readable medium for storing instructions causing one or more processors to perform the method described in any one of claims 1 to 29.

32. An apparatus comprising at least one processor and at least one non-temporary computer-readable medium for storing instructions causing the at least one processor to perform the method according to any one of claims 1 to 29.

33. A computer-readable medium characterized by storing an encoded information scene description file generated according to the method described in any one of claims 16 to 29.

34. A signal comprising a scene description file generated according to the method described in any one of claims 1 to 29.

Images