Carrying coded haptic data in a media container

Abstract

Systems and methods are described for encoding, processing, and / or decoding container files, such as ISOBMFF container files, that represent haptic data. A method according to some embodiments includes: obtaining a container file that includes a plurality of haptic tracks, the container file including information associating each of the plurality of haptic tracks with at least one of a respective device, a respective perception, or a respective avatar; obtaining information indicating a selection of at least one device, at least one perception, or at least one avatar; and extracting haptic data in response to the selection, the extracted haptic data excluding at least one of the plurality of haptic tracks that is not associated with any selected device, perception, or avatar.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application is a nonprovisional application of U.S. Provisional Patent Application No. 63 / 417,638, entitled "Carriage of Coded Haptics Data in Media Containers," filed October 19, 2022, and claims the benefit under 35 U.S.C. §119(e), which is incorporated herein by reference in its entirety. [Background technology]

[0002] A haptic sequence is a set of data encoded for rendering based on the sense of touch and position in space, similar to how a video sequence is a set of encoded data for rendering using vision. A haptic sequence encodes, for example, temporal data represented as tracks associated with a haptic device. A haptic device may render different modalities of the sense of touch and position in space, such as vibration, force, position, velocity, or temperature.

[0003] A new standard (ISO / IEC 23090-31) is currently under development by the Motion Picture Experts Group (MPEG). Figure 2 shows the MPEG haptic codec architecture. In this architecture, the coded representation of haptic data can be in one of two formats: an interchange format (.hjif) or a distribution format (.hmpg). The interchange format is a JSON-based, human-readable description of the haptic data, while the distribution format is a compressed binary representation of the data. The two formats have complementary purposes, and lossless or lossy one-to-one conversion can be performed between them. The compressed binary bitstream of the distribution format is structured into a sequence of Network Abstraction Layer (NAL) units to facilitate encapsulation by any network protocol or file format.

[0004] The haptic decoder takes as input a binary .hmpg or .hjif file and outputs an .hjif file. The haptic data contained in the resulting .hjif file can then be rendered directly on a haptic device or using an intermediate synthesizer that generates pulse code modulation (PCM) data.

[0005] Figure 2 is a functional block diagram showing a haptic codec architecture corresponding to ISO / IEC 23090-31. The data structure of the two codec formats follows a hierarchical organization shown in Figure 3.

[0006] The highest level of structure describes the entire haptic experience defined in the file or stream. It includes some high-level metadata information. It also provides a list of avatars (body representations) that can be referenced to specify the desired location of haptic stimulation on the body. The haptic data itself is described through a list of "sensations." These sensations correspond to haptic signals associated with a particular sensory modality (vibration, force, position, velocity, temperature, etc.).

[0007] In addition to the perception-specific metadata, a perception also contains a list of channels. Within a channel, the data is broken down into frequency bands. Each band defines a portion of the signal within a given frequency range. The bands are described by a list of haptic effects, each of which contains a list of keyframes. The haptic signal within a channel can then be reconstructed by combining the data in different bands (adding high-frequency and low-frequency bands).

[0008] Figure 4 shows an example of a Network Abstraction Layer (NAL) unit structure in a haptic bitstream, and Figure 5 shows NAL unit payload types.

[0009] The ISO / IEC 14496 (MPEG-4) standard contains several sections that define file formats for the storage of time-based media. All of these are based on and derived from the ISO Base Media File Format (ISOBMFF), a structural, media-independent definition. ISOBMFF primarily contains structure and media data information for the timed presentation of media data, such as audio and video. There is also support for non-timed data, such as metadata, at different levels within the file structure. The logical structure of a file is a "movie," which then contains a set of time-parallel "tracks." The temporal structure of the file is that tracks contain sequences of temporal "samples," which are mapped to the overall movie timeline. ISOBMFF is based on the concept of a box-structured file. A box-structured file consists of a series of boxes (sometimes called atoms) with a size and a type. The type is a 32-bit value, typically chosen to be four printable characters, also known as a four-character code (4CC). Non-timed data may be included in metadata boxes at the file level or attached to one of the streams of timed data, called movie boxes or tracks within a movie.

[0010] Among the top-level boxes in an ISOBMFF container is a MovieBox ("moov"), which contains metadata for the continuous media streams present in the file. These metadata are signaled within a hierarchy of boxes within the Movie box, such as TrackBoxes ("trak"). Tracks represent the continuous media streams present in the file. The media streams themselves consist of a sequence of samples, such as audio or video access units of elementary media streams, and are enclosed within a MediaDataBox ('mdat'), which resides at the top level of the container. The metadata for each track contains a list of sample description entries, each of which provides the coding or encapsulation format used in the track and initialization data for processing that format. Each sample is associated with one of the track's sample description entries. ISO / IEC 14496-12 provides tools for defining an explicit timeline map for each track. This is known as an edit list and is signaled using an EditListBox with the following syntax, where each entry defines a portion of the track timeline by mapping a portion of the composition timeline or by indicating "empty" time (a portion of the presentation timeline that does not map to media, an "empty" edit).

number

[0011] Systems and methods are described for encoding and / or decoding container files, such as ISOBMFF container files, representing haptic data. In one example, a haptic experience track is encoded within the container file. The haptic experience track includes information describing at least one available avatar for the haptic experience and configuration information for at least one perception in the haptic experience. The haptic experience track may further reference at least one haptic track. The haptic track may include all bands of channels for each one of the perceptions. The haptic track may have samples carrying haptic band data bitstream units.

[0012] Encoder and decoder devices are provided for performing the methods described herein. The encoder or decoder device may include a processor configured to perform the methods described herein. The device may include a computer-readable medium (e.g., a non-transitory medium) that stores instructions for performing the methods described herein. In some embodiments, the computer-readable medium (e.g., a non-transitory medium) stores haptic data encoded using any of the methods described herein.

[0013] A method according to some embodiments includes obtaining a container file, such as an ISOBMFF file, containing a plurality of haptic tracks, the container file including information associating each of the plurality of haptic tracks with at least one of a respective device, a respective perception, or a respective avatar; obtaining information indicating a selection of at least one device, at least one perception, or at least one avatar; and extracting haptic data in response to the selection, the extracted haptic data excluding at least one of the plurality of haptic tracks not associated with any selected device, perception, or avatar.

[0014] In some embodiments, the method is performed by a server and the information indicative of the selection is received from the client device.

[0015] Some embodiments further include providing the extracted haptic data in a bitstream to a client device.

[0016] Some embodiments further include providing a manifest file indicating at least one available device, at least one available perception, or at least one available avatar, and information indicating the selection is received in response to the manifest file.

[0017] Some embodiments further include providing the extracted haptic data to the client device as a container file.

[0018] Some embodiments further include rendering the extracted haptic data.

[0019] In some embodiments, the information associating a haptic track with a respective device includes a device identifier in a haptic channel configuration box associated with the respective track.

[0020] In some embodiments, the information associating a haptic track with each perception includes information identifying a track group that includes multiple tracks associated with each perception.

[0021] In some embodiments, the information associating a haptic track with a respective avatar includes an avatar identifier in a haptic perception configuration box.

[0022] In some embodiments, the container file includes information associating each of a plurality of haptic tracks with a respective device, the selection being a selection of at least one device, and the extracted haptic data excludes at least one of the plurality of haptic tracks that is not associated with any selected device.

[0023] In some embodiments, an apparatus having one or more processors is configured to at least: obtain a container file, such as an ISOBMFF file, including a plurality of haptic tracks, the container file including information associating each of the plurality of haptic tracks with at least one of a respective device, a respective perception, or a respective avatar; obtain information indicating a selection of at least one device, at least one perception, or at least one avatar; and extract haptic data in response to the selection, the extracted haptic data excluding at least one of the plurality of haptic tracks that is not associated with any selected device, perception, or avatar.

[0024] One or more of the present embodiments also provide a computer-readable storage medium having stored thereon instructions for performing any of the methods described herein. Some embodiments include a computer-readable storage medium having stored thereon a bitstream or container file generated according to the methods described herein. Some embodiments include a computer program product including instructions for performing any of the methods described herein. [Brief explanation of the drawings]

[0025] [Figure 1A] 1 is a system diagram illustrating an example communication system in which one or more disclosed embodiments may be implemented. [Figure 1B] 1B is a system diagram illustrating an example wireless transmit / receive unit (WTRU) that may be used within the communication system shown in FIG. 1A, according to an embodiment. [Figure 1C] FIG. 1 is a functional block diagram of a system used in some embodiments described herein. [Figure 2] 1 shows the MPEG haptic codec architecture. [Figure 3] The hierarchical data structure of the two codec formats is shown. [Figure 4]1 shows the NAL unit structure in a haptic bitstream. [Figure 5] Indicates the NAL unit payload type. [Figure 6] 1 illustrates a method performed by a server in some embodiments. [Figure 7] 1 illustrates a method performed by a client in some embodiments.

[0026] Exemplary Network for Implementation of the Embodiments 1A illustrates an example communication system 100 in which one or more disclosed embodiments can be implemented. The communication system 100 may be a multiple-access system that provides content, such as voice, data, video, messaging, broadcasts, and the like, to multiple wireless users. The communication system 100 may enable the multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communication system 100 may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tailed unique word DFT-spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block filtered OFDM, filter bank multicarrier (FBMC), and the like.

[0027] 1A, communications system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, and 102d, a RAN 104, a CN 106, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, although it will be understood that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of WTRUs 102a, 102b, 102c, and 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d (any of which may be referred to as a “station” and / or “STA”) may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, mobile phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearables, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in the context of industrial and / or automated processing chains), consumer electronics devices, devices operating in commercial and / or industrial wireless networks, etc. Any of the WTRUs 102a, 102b, 102c, and 102d may be referred to interchangeably as a UE.

[0028] The communications system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communications networks, such as the CN 106, the Internet 110, and / or other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node-B, an eNode-B, a home Node-B, a home eNode-B, a gNB, an NR Node-B, a site controller, an access point (AP), a wireless router, etc. Although the base stations 114a, 114b are each shown as a single element, it will be understood that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0029] The base station 114a may be part of the RAN 104, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide wireless service coverage for a particular geographic area, which may be relatively fixed or may change over time. A cell may be further divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e., one transceiver for each sector of the cell. In one embodiment, the base station 114a may employ multiple-input multiple output (MIMO) technology and may utilize multiple transceivers per sector of the cell, for example, using beamforming to transmit and / or receive signals in desired spatial directions.

[0030] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0031] More specifically, as noted above, the communications system 100 may be a multiple-access system, but may use one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, the base station 114a and the WTRUs 102a, 102b, 102c of the RAN 104 may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 116 using wideband CDMA (WCDMA). WCDMA may include communications protocols such as High-Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High-Speed ​​Downlink Packet Access (HSDPA) and / or High-Speed ​​Uplink Packet Access (HSUPA).

[0032] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro).

[0033] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as NR radio access, which may establish the air interface 116 using New Radio (NR).

[0034] In one embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may jointly implement LTE radio access and NR radio access, e.g., using a dual connectivity (DC) principle. Thus, the air interface utilized by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to and from multiple types of base stations (e.g., eNBs and gNBs).

[0035] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement a wireless technology such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi)), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rates for GSM Evolution (EDGE), GSM EDGE (GERAN), or the like.

[0036] 1A may be, for example, a wireless router, a Home NodeB, a Home eNodeB, or an access point, but may utilize any suitable RAT to facilitate wireless connectivity in a local area such as a location such as a business, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a road, etc. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may establish a picocell or a femtocell using a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.). As shown in FIG. 1A, the base station 114b may be directly connected to the Internet 110. Thus, the base station 114b may not need to access the Internet 110 through the CN 106.

[0037] The RAN 104 may communicate with the CN 106, which may be any type of network configured to provide voice, data, application, and / or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have various quality of service (QoS) requirements, such as different throughput, latency, error tolerance, reliability, data throughput, mobility, etc. The CN 106 may provide call control, billing services, mobile location-based services, prepaid calling, Internet connectivity, video distribution, etc., and / or perform high-level security functions such as user authentication. Although not shown in FIG. 1A , it will be understood that the RAN 104 and / or CN 106 may communicate directly or indirectly with other RANs that use the same RAT as the RAN 104 or a different RAT. For example, in addition to being connected to the RAN 104, which may utilize NR radio technology, the CN 106 may also communicate with another RAN (not shown) employing GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0038] The CN 106 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network providing plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the transmission control protocol (TCP), the user datagram protocol (UDP), and / or the internet protocol (IP) of the TCP / IP Internet protocol suite. The networks 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, the network 112 may include another CN connected to one or more RANs, which may use the same RAT as the RAN 104 or a different RAT.

[0039] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links.) For example, the WTRU 102c shown in FIG. 1A may be configured to communicate with a base station 114a that may employ a cellular-based wireless technology and a base station 114b that may employ an IEEE 802.2 wireless technology.

[0040] 1B is a system diagram illustrating an example WTRU 102. As shown in FIG. 1B, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138. It will be understood that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0041] The processor 118 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. While FIG. 1B depicts the processor 118 and the transceiver 120 as separate components, it will be understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0042] The transmit / receive element 122 may be configured to transmit or receive signals to or from a base station (e.g., base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In one embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive IR signals, UV signals, or visible light signals, for example. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF signals and light signals. It will be understood that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0043] 1B as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0044] The transceiver 120 may be configured to modulate signals transmitted by the transmit / receive element 122 and demodulate signals received by the transmit / receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as, for example, NR and IEEE 802.11.

[0045] The processor 118 of the WTRU 102 may be coupled to and may receive user-entered data from a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light-emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. Additionally, the processor 118 may access information from and store data in any type of suitable memory, such as non-removable memory 130 and / or removable memory 132. The non-removable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, etc. In other embodiments, the processor 118 may access information from and store data in memory that is not physically located on the WTRU 102, such as on a server or home computer (not shown).

[0046] The processor 118 may receive power from the power source 134 and may be configured to distribute and / or control the power to other components in the WTRU 102. The power source 134 may be any suitable device for providing power to the WTRU 102. For example, the power source 134 may include one or more dry batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0047] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or instead of, information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) over the air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be appreciated that the WTRU 102 may obtain location information by way of any suitable location determination method while remaining consistent with an embodiment.

[0048] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or videos), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, etc. The peripheral device 138 may include one or more sensors, which may be one or more of a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, a direction sensor, a proximity sensor, a temperature sensor, a time sensor, a geolocation sensor, an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0049] The WTRU 102 may include a full-duplex radio for transmitting and receiving some or all of the signals (e.g., associated with a particular subframe) on both the UL (e.g., for transmission) and downlink (e.g., for reception) in parallel and / or simultaneously. The full-duplex radio may include an interference management unit for reducing and / or substantially eliminating self-interference through either hardware (e.g., a choke) or signal processing via a processor (e.g., a separate processor (not shown) or via processor 118). In one embodiment, the WTRU 102 may include a half-duplex radio for transmitting and receiving some or all of the signals (e.g., associated with a particular subframe on either the UL (e.g., for transmission) or downlink (e.g., for reception) in parallel and / or simultaneously.

[0050] Although the WTRU is depicted in FIGS. 1A-1B as a wireless terminal, it is contemplated that in certain representative embodiments such a terminal may use a wired communication interface (e.g., temporarily or permanently) with the communication network.

[0051] In an exemplary embodiment, the other network 112 may be a WLAN.

[0052] 1A-1B and in view of the corresponding description, one or more or all of the functions described herein may be performed by one or more emulation devices (not shown). An emulation device may be one or more devices configured to emulate one or more or all of the functions described herein. For example, the emulation device may be used to test other devices and / or simulate network and / or WTRU functions.

[0053] The emulation device may be designed to implement one or more tests of other devices in a lab environment and / or an operator network environment. For example, one or more emulation devices may perform one or more or all functions while fully or partially implemented and / or deployed as part of a wired and / or wireless communication network to test other devices in the communication network. One or more emulation devices may perform one or more or all functions while temporarily implemented / deployed as part of a wired and / or wireless communication network. The emulation device may be directly coupled to another device for testing purposes and / or may use over-the-air wireless communication to perform the tests.

[0054] One or more emulation devices may perform one or more functions, inclusive, while not being implemented / deployed as part of a wired and / or wireless communication network. For example, the emulation devices may be utilized in test scenarios in a test lab and / or in an undeployed (e.g., test) wired and / or wireless communication network to implement testing of one or more components. One or more emulation devices may be test equipment. Direct RF coupling and / or wireless communication via RF circuitry (e.g., which may include one or more antennas) may be used by the emulation devices to transmit and / or receive data. Exemplary System.

[0055] The embodiments described herein are not limited to being implemented on a WTRU. Such embodiments may be practiced using other systems, such as the system of FIG. 1C. FIG. 1C illustrates a block diagram of an example system in which various aspects and embodiments may be implemented. System 1000 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television sets, personal video recording systems, connected home appliances, and servers. The elements of system 1000, alone or in combination, may be embodied in a single integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or separate components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports, and is configured to implement one or more of the aspects described herein.

[0056] The system 1000 includes at least one processor 1010 configured to execute loaded instructions, for example, to implement various aspects described herein. The processor 1010 can include embedded memory, input / output interfaces, and various other circuitry known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which can include non-volatile memory and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drives, and / or optical disk drives. Storage devices 1040 may include, by way of non-limiting example, internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0057] System 1000 includes an encoder / decoder module 1030 configured to process data to provide, for example, encoded video or decoded video, which may include its own processor and memory. Encoder / decoder module 1030 represents a module that may be included in a device that performs encoding and / or decoding functions. As is known, a device may include one or both of an encoding and a decoding module. Additionally, encoder / decoder module 1030 may be implemented as a separate element of system 1000 or may be incorporated within processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0058] Program code loaded into the processor 1010 or the encoder / decoder 1030 to perform various aspects described herein may be stored in the storage device 1040 and subsequently loaded into the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing of equations, expressions, operations, and operational logic.

[0059] In some embodiments, memory internal to the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be either the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be the memory 1020 and / or the storage device 1040, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, for example, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, a high-speed external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations such as MPEG-2 (MPEG stands for Moving Picture Experts Group, MPEG-2 is also referred to as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, an emerging standard developed by JVET, i.e., the Joint Video Experts Team).

[0060] Inputs to the elements of system 1000 may be provided through various input devices, as shown in block 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcast station, (ii) a component (COMP) input terminal (or set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and / or (iv) a High-Definition Multimedia Interface (HDMI) input terminal. Other examples not shown in FIG. 1C include composite video.

[0061] In various embodiments, the input devices of block 1130 have associated respective input processing elements as known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or bandlimiting a signal to a frequency band), (ii) downconverting the selected signal, (iii) bandlimiting again to a narrower frequency band to select a signal frequency band, which in certain embodiments may be referred to as a channel (for example), (iv) demodulating the downconverted, bandlimited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements that perform these functions, e.g., a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may include, for example, a tuner that performs various of these functions, including downconverting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency near baseband) or to baseband. In one set-top box embodiment, the RF section and its associated input processing elements perform frequency selection by receiving, filtering, downconverting, and filtering again to the desired frequency band an RF signal transmitted over a wired (e.g., cable) medium. In various embodiments, the order of these (and other) elements is rearranged, some of these elements are removed, and / or other elements that perform similar or different functions are added. Adding elements can include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0062] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or within processor 1010, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented, as desired, within a separate interface IC or within processor 1010. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 1010 and an encoder / decoder 1030, which operates in combination with memory and storage elements to process the data stream as desired for display on an output device.

[0063] The various elements of system 1000 may be provided within a unitary housing in which the various elements are interconnected and capable of transmitting data between them using a suitable connection arrangement 1140, e.g., an internal bus as known in the art, including an Inter-IC (I2C) bus, wiring, and printed circuit boards.

[0064] System 1000 includes a communication interface 1050 that enables communication with other devices over a communication channel 1060. Communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 1060. Communication interface 1050 may include, but is not limited to, a modem or a network card, and communication channel 1060 may be implemented in a wired and / or wireless medium, for example.

[0065] In various embodiments, data is streamed or otherwise provided to system 1000 using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in these embodiments is received via communication channel 1060 and communication interface 1050, which are adapted for Wi-Fi communication. Communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, enabling streaming applications and other over-the-top communications. Other embodiments provide streamed data to system 1000 using a set-top box that delivers data via an HDMI connection in input block 1130. Still other embodiments provide streamed data to system 1000 using an RF connection in input block 1130. As noted above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0066] System 1000 can provide output signals to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. Display 1100 in various embodiments includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 1100 may be for a television, a tablet, a laptop, a mobile phone, or other device. Display 1100 may also be integrated with other components (e.g., as in a smartphone) or may be separate (e.g., an external monitor for a laptop). Other peripheral devices 1120, in various example embodiments, include one or more of a standalone digital video disc (or digital versatile disc) (both terms DVR), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 to provide functionality based on the output of system 1000. For example, a disc player performs the function of playing the output of system 1000.

[0067] In various embodiments, control signals are communicated between system 1000 and display 1100, speakers 1110, or other peripheral devices 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable inter-device control with or without user intervention. Output devices may be communicatively coupled to system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, output devices may be connected to system 1000 using communication channel 1060 via communication interface 1050. Display 1100 and speakers 1110 may be integrated into a single unit with other components of system 1000 in an electronic device such as a television. In various embodiments, display interface 1070 includes a display driver, such as a timing controller (TCon) chip.

[0068] Display 1100 and speakers 1110 may alternatively be separate from one or more of the other components, for example, if the RF portion of input 1130 is part of a separate set-top box. In various embodiments in which display 1100 and speakers 1110 are external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0069] The embodiments may be implemented by the processor 1010, or by computer software implemented by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type appropriate to the technical environment and may be implemented using any suitable data storage technology, such as, by way of non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 1010 may be of any type appropriate to the technical environment and may include, by way of non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture. DETAILED DESCRIPTION OF THE INVENTION

[0070] Problems addressed in some embodiments. While the bitstream format being developed in ISO / IEC 23090-31 allows for describing haptic experiences in a compact representation that can be easily consumed by a haptic decoder, the bitstream format does not allow scalable access to different stream components and does not allow for selective streaming. Furthermore, immersive experiences involve several different media types that are synchronized during playback to provide a targeted experience. Currently, there is no well-defined way to store and multiplex such haptic bitstreams within an ISOBMFF media container along with other types of media, such as audio and video.

[0071] This disclosure describes a versatile and scalable design for supporting the transport of haptic bitstreams generated by the ISO / IEC 23090-31 haptic codec in an ISOBMFF container. Example embodiments can be applied to an ecosystem including coding of immersive media, storage and streaming of coded haptic media content, and decoding of haptic media data on devices or any service that provides an immersive media experience.

[0072] Tactile experience track. In an exemplary embodiment, the main entry into a haptic experience is a haptic experience track. A haptic experience track may have a sample entry of type HapticsSampleEntry with a "hexp" four-character code (4CC). A HapticsSampleEntry with a "hexp" 4CC contains a HapticsExperienceConfigurationBox that describes the configuration of the haptic experience, including a description of the available avatars in the experience and the configuration of the various perceptions therein. A haptic experience track may reference one or more haptic tracks. Each haptic track may carry band data for a particular channel of a particular perception in the haptic experience.

[0073] To link a haptic experience track with related haptic tracks, the track reference tools of ISO / IEC 14496-12 can be used. A TrackReferenceTypeBox with reference type "hpbd" can be added to a TrackReferenceBox within the TrackBox of a haptic experience track. The TrackReferenceTypeBox can contain an array of track_IDs that specify the identifiers of the referenced haptic tracks.

[0074] In another embodiment, each haptic track carries all bands belonging to one particular channel of perception of the haptic experience.

[0075] The HapticsSampleEntry and HapticsExperienceConfigurationBox may have the following structure:

number

number

number

[0076] In one example, the semantics of the fields of the HapticsExperienceConfigurationBox are as follows: ● creation_time is an integer that declares the creation time of this haptic experience (in UTC time, seconds since midnight on January 1, 1904). ● Description is a human-readable description of the experience.

[0077] In one example, the semantics of the HapticsAvatarDescriptionBox field are as follows: ● avatar_count is the number of avatars available in the haptic experience. ● avatar_id is a unique identifier for the avatar. ● level_of_detail indicates the level of detail of the avatar. The 3D mesh corresponding to the avatar provides high resolution and precision with a variable vertex density depending on the application. Therefore, a particular mesh resolution may be related to the tactile spatial clarity of the associated haptic perception. ● avatar_type indicates the type of haptic perception represented by the avatar based on the spatial clarity of the corresponding haptic modality. Examples of possible values ​​are shown in Table 1. [Table 1] ● mesh_uri provides a URI to access the avatar's associated custom mesh file. It shall only be present if the value of avatar_type is 0.

[0078] avatar_type specifies the type of haptic sensation represented by the avatar. Values ​​can be vibration, pressure, temperature, or custom. For a "custom" mesh, the mesh is provided as a companion file. If the type is one of the first three, the avatar mesh may be predefined for applications targeting a specific modality. If you have multiple modalities, a custom avatar is likely the best option.

[0079] In one example, the HapticsPerceptionConfigurationBox has the following syntax:

number

number

number

[0080] In one example, the semantics of the HapticsPerceptionConfigurationBox field are as follows: ● perception_count is the number of perceptions in the tactile experience. ● perception_id is a unique identifier for the perception. ● description is a human-readable description of the perception. ● modality indicates the type of perception. Examples of possible values ​​are shown in Table 2. [Table 2] ● avatar_id is the avatar identifier of the avatar body model associated with the perception. ● unit_exponent is 10 ^ Refers to the x exponent in SI units that identifies the representation space of the independent variable. ● perception_unit_exponent refers to the 10^x exponent of the SI unit measure of the dependent variable ● channels_count is the number of haptic channels for perception. ● devices_count is the number of haptic reference devices associated with the perception. ● device_id is a unique identifier for the device. ● device_type indicates the type of actuator. Possible values ​​are listed in Table 3. [Table 3] ● body_part_mask is a binary mask that specifies the location of the device or actuator on the body. The semantics of each bit in the bit mask in one example may be described in the table below. [Table 4] ● device_mask is the bitmask associated with the device. ● maximum_frequency indicates the maximum frequency of the actuator in Hertz (Hz). ● minimum_frequency indicates the maximum frequency of the actuator in Hertz (Hz). ● resonance_frequency indicates the resonant frequency of the actuator in Hertz (Hz). ● maximum_amplitude indicates the maximum amplitude value of the target device according to the perceptual modality. ● Impedance indicates the impedance of the actuator in ohms (Ω). ● maximum_voltage indicates the maximum voltage of the actuator in volts (V). ● maximum_current indicates the maximum current in amperes (A) for the actuator. ● maximum_displacement indicates the maximum displacement of the actuator in millimeters (mm). ● weight indicates the weight of the device in kilograms (Kg). ● size indicates the size of the device in millimeters (mm). ● user_defined_data can be used to specify additional properties of the target device.

[0081] In another embodiment, the HapticsPerception data structure is defined as follows and includes another dedicated ISOBMFF structure for signaling information about the reference device:

number

[0082] In one example, the HapticsReferenceDevicesBox may be defined as follows:

number

[0083] In one example, the semantics of the fields of the HapticsReferenceDevicesBox may be as follows: ● devices_count is the number of haptic reference devices associated with the perception.

[0084] In some embodiments, the HapticsEffectsLibraryBox contains an optional box that may be present when an effects library is present in the haptic perception, and the band data unit may reference effects in this library instead of keeping the effect definitions in the band data. The HapticsEffectsLibraryBox may be configured as follows:

number

number

number

[0085] In one example, the semantics of the fields of HapticsEffectsLibraryBox are as follows: ● effects_count is the number of effects in the effects library. ● effect is an instance of HapticsLibraryEffect. ● effect_id indicates the unique identifier of the library effect. ● effect_type indicates the type of effect. Examples of acceptable values ​​are shown in the table below. [Table 5] Table 4. Haptic effect types • Position indicates the temporal position of the effect relative to the onset of the experience. ● Phase indicates the phase of the effect. ● band_type indicates the type of data in the band. Examples of possible values ​​are shown in Table 5. [Table 6] Table 5. Band types ● curve_type indicates the interpolation function for bands of type "curve". It is present only if band_type has the value 1. Examples of possible values ​​are shown in Table 6. [Table 7] Table 6. Curved band types ● base_signal_type indicates the type of base signal. Examples of possible values ​​are shown in Table 7. [Table 8] Table 7. Base Signal Types ● timeline_effects_count indicates the number of effects available in the timeline for this band. ● keyframe_id is a unique identifier for the effect's keyframe. ● amplitude is the amplitude of the keyframe. ● position is the relative position of the keyframe. ● frequency is the relative frequency of the keyframes.

[0086] In another embodiment, the HapticsEffectsLibraryBox may be defined as follows:

number

[0087] The semantics of the fields of the HapticsEffectsLibraryBox in this embodiment may be as follows: ● band_data_unit is an instance of a bitstream data unit that carries band data. Tactile track.

[0088] According to some embodiments, a haptic track is provided as a track whose samples carry haptic band data bitstream units. Like a haptic experience track, a haptic track also contains a HapticsSampleEntry, but with a different 4CC type. Two different 4CCs may be used to distinguish between a haptic track whose samples carry data for a single band of a sensory-channel combination and a haptic track whose samples carry data for all bands of a particular sensory channel. For example, a "hpd1" 4CC may be used for the former, and a "hpd2" 4CC may be used for the latter.

[0089] In some embodiments, the definition of a HapticsSampleEntry of type "hpd1" is as follows:

number

[0090] In some embodiments, the definition of HapticsChannelConfigurationBox is as follows:

number

[0091] The corresponding semantics of the fields of the HapticsChannelConfigurationBox may be as follows: ● channel_id is a unique identifier for the channel. ● direction_present_flag is a flag that indicates whether a direction is associated with this channel. ● device_id is the identifier of the reference device associated with this channel. Gain indicates the gain associated with the channel to adapt the normalized encoded data values ​​to a typical device. ● mixing_weight indicates the weight of the channels when mixing the different channels together to produce the final signal. ● body_part_mask is a bitmask that indicates which body parts are affected. ● frequency_sampling indicates the sampling frequency of the original encoded signal in Hertz (Hz). ● samples_count indicates the number of samples in the original coded signal. This field is present if the frequency_sampling value is greater than 0. ● bands_count is the number of bands available for this channel. ● direction_x indicates the horizontal direction of left and right in local space. ● direction_y indicates the vertical direction of the local space. ● direction_z indicates the forward / backward direction in local space. ● vertices_count is the number of avatar vertices affected by the effect.

[0092] In some embodiments, the definition of HapticsBandConfigurationBox is as follows:

number

[0093] In one example, the semantics of the fields of the HapticsBandConfigurationBox may be as follows: ● band_id is a unique identifier for the band. ● band_type indicates the type of band. Examples of possible values ​​are shown in Table 8. [Table 9] Table 8. Band types ● curve_type indicates the interpolation function to be used when band_type has a value of 1 (i.e., curve bands). ● window_length is the duration of the haptic keyframe. ● freq_low is the lower frequency limit of the band. ● freq_high is the upper frequency limit of the band. Tactile track sample.

[0094] For a haptic track, depending on the sample entry type as defined by the 4CC of the sample entry, the samples of this track may carry data for either only one band of a perceptual channel, or all bands of a perceptual channel. If all bands are carried in a sample of a haptic track, each sample may be composed of several sub-samples, each sub-sample containing data for one of the bands.

[0095] In another embodiment, keyframes of a haptic effect for a channel band may be stored in separate samples within a haptic track, with each sample assigned a decoding timestamp and a synthesis timestamp corresponding to the keyframe's relative position with respect to the start of the haptic experience. To identify samples belonging to an effect, a number of sample groups may be defined in the metadata of the ISOBMFF track. Each sample containing data for an independent keyframe, e.g., the first keyframe of an effect, may be designated as a synchronization sample, allowing random access within the track. Grouping tactile tracks for tactile perception.

[0096] When multiple haptic tracks are used to carry band data for various channels of haptic experience perception, in some embodiments, track grouping can be used to identify which haptic track is associated with a particular haptic perception.

[0097] This can be done in some embodiments by defining a track group type that extends the TrackGroupTypeBox defined in ISO / IEC 14496-12, which includes a track_group_id field that represents an identifier for the track group, and a track_group_type field that stores a four-character code that identifies the group type. The track_group_id and track_group_type pair identify a track group within a container file.

[0098] An example HapticsTrackGroupBox may be defined as follows, in some embodiments:

number

[0099] Using the systems, methods, and data structures described herein, a player can extract only the bitstream elements it needs, for example, only those associated with a particular avatar, or those belonging to a particular perception. This can be particularly useful in the context of streaming, where exposure can be contained in a manifest file.

[0100] In contrast to systems that use only a single track to carry the haptic bitstream, exemplary embodiments enable the use of a multi-track design, where data belonging to a particular channel and / or perception is carried in its own track. In some embodiments, all of the band data for an effect is provided in one sample. Some embodiments enable a parser to extract a band from a sample by using sub-samples. The systems and methods described herein enable container files to be structured in a way that makes it easier for a player to extract only the data it needs.

[0101] In an exemplary method as shown in the flowchart of FIG. 6 , a method performed in some embodiments includes obtaining 602 a container file including a plurality of haptic tracks, the container file including information associating each of the plurality of haptic tracks with at least one of a respective device, a respective perception, or a respective avatar. At 604, information indicating a selection of at least one device, at least one perception, or at least one avatar is obtained. At 606, haptic data is extracted in response to the selection, the extraction being performed to exclude at least one of the plurality of haptic tracks that is not associated with any of the selected devices, perceptions, or avatars. For example, the extracted haptic data may include all tracks associated with at least one selected device, perception, or avatar, and in some embodiments, all remaining haptic tracks are excluded from the extracted data. In some embodiments, the extracted haptic data may include all tracks associated with the selected device, selected perception, and selected avatar, and in some embodiments, all remaining haptic tracks are excluded from the extracted data.

[0102] 6 may be performed by a server, and the selection information received at 604 may be received from a client device. In some such embodiments, the extracted haptic data may be provided to the client device in a bitstream. In some such embodiments, the server optionally provides a manifest file at 610 indicating at least one device, at least one available perception, or at least one available avatar, and the information indicating the selection is received in response to the manifest file.

[0103] As an example, a server may obtain a container file having a first plurality of haptic tracks associated with a first perception and a second plurality of haptic tracks associated with a second perception. The first perception may include information indicating that the modality of the first perception is “wind.” The second perception may include information indicating that the modality of the second perception is “vibrotactile.” The server may use metadata in the container file to provide (e.g., at 610) a manifest file having information characterizing the first perception and the second perception. In response to the manifest file, the client may request only the second perception (vibrotactile) and not the first perception (wind). For example, the client may not have a device capable of rendering the sensation of “wind,” or the user may prefer not to experience simulated wind. In response to the client selection, the haptic data provided to the client excludes data based on the tracks associated with the first perception. Similar selections may be made based on available avatars and / or available devices. For example, a client may not have some types of haptic devices, and therefore only haptic data relevant to the devices that the client does have may be streamed to the client.

[0104] In an exemplary method as shown in the flowchart of FIG. 7 , a method performed in some embodiments includes, at 702, obtaining a container file including a plurality of haptic tracks, the container file including information associating each of the plurality of haptic tracks with at least one of a respective device, a respective perception, or a respective avatar. At 704, information indicating a selection of at least one device, at least one perception, or at least one avatar is obtained. This information may include configuration information regarding the types of haptic devices available to the user and / or the types of haptic experiences the user prefers to enjoy or avoid. At 706, haptic data is extracted in response to the selection, the extraction being performed to exclude at least one of the plurality of haptic tracks that is not associated with any of the selected devices, perceptions, or avatars. At 708, the client can render the extracted haptic data using an appropriate actuator. Further embodiments.

[0105] A method according to some embodiments includes encoding a haptic experience track within a container file, the haptic experience track including information describing at least one available avatar for the haptic experience and configuration information for at least one perception in the haptic experience.

[0106] A method according to some embodiments includes decoding a haptic experience track from a container file, the haptic experience track including information describing at least one available avatar for the haptic experience and configuration information for at least one perception in the haptic experience.

[0107] In some embodiments, the haptic experience track further references at least one haptic track, for example, the haptic experience track may include an array of identifiers of each reference haptic track.

[0108] In some embodiments, each of the haptic tracks includes all bands of channels for a respective one of the senses.

[0109] In some embodiments, a haptic track includes band data for at least one sensory channel.

[0110] In some embodiments, a haptic track is a track having samples that carry haptic band data bitstream units.

[0111] In some embodiments, the haptic track includes a code indicating whether (i) the haptic track includes samples carrying data for a single band of a sensory-channel combination, or (ii) the haptic track includes samples carrying data for all bands of a particular sensory channel.

[0112] In some embodiments, the samples of the haptic track contain data for only one band of a sensory channel.

[0113] In some embodiments, the samples of the haptic track include data for all bands of the sensory channel, hi some such embodiments, each sample includes multiple sub-samples, each sub-sample including data for one of the bands.

[0114] In some embodiments, the plurality of samples are keyframe samples.

[0115] In some embodiments, multiple keyframe samples are designated as synchronization samples.

[0116] In some embodiments, the file includes metadata defining at least one sample group, and the samples in a common sample group are samples that belong to a specified effect.

[0117] In some embodiments, the file further includes haptic track group information that associates multiple haptic tracks with corresponding haptic track groups.

[0118] This disclosure describes a wide variety of aspects, including tools, features, embodiments, models, approaches, and the like. Many of these aspects are described specifically and may be described in a definitive manner to at least indicate their individual characteristics. However, this is for clarity of description and does not limit the disclosure or scope of these aspects. In fact, all of the different aspects can be combined and interchanged to provide additional aspects. Furthermore, these aspects can also be combined and interchanged with aspects described in previous applications.

[0119] Aspects described and contemplated in this disclosure can be implemented in many different formats. While some embodiments are specifically illustrated, other embodiments are contemplated, and discussion of a particular embodiment is not intended to limit the breadth of implementations. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting generated or encoded bitstreams. These and other aspects can be embodied as a method, an apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0120] In this disclosure, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image," "picture," and "frame" may be used interchangeably. Typically, but not necessarily, the term "reconstructed" is used on the encoder side, while "decoded" is used on the decoder side.

[0121] Various methods are described herein, each of which includes one or more steps or acts for achieving the described method. Unless a specific order of steps or acts is required for the successful operation of the method, the order and / or use of specific steps and / or acts may be modified or combined. Furthermore, terms such as “first,” “second,” etc. may be used in various embodiments to modify elements, components, steps, operations, etc., e.g., “first decode” and “second decode.” The use of such terms does not imply a modified order of operations unless specifically required. Thus, in this example, the first decode need not be performed before the second decode, but could occur, for example, before, during, or during an overlapping period with the second decode.

[0122] For example, various numerical values ​​may be used in this disclosure. The specific values ​​are for illustrative purposes, and the described aspects are not limited to these specific values.

[0123] The embodiments described herein may be performed by computer software implemented by a processor or other hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The processor may be of any type appropriate to the technical environment, and may include, by non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0124] Various implementations involve decoding. As used in this disclosure, "decoding" can encompass, for example, all or part of the processes performed on a received encoded sequence to generate a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by decoders in various implementations described in this disclosure, such as extracting an image from a tiled (packed) image, determining an upsample filter to use, then upsampling the image, and flipping the image back to its intended orientation.

[0125] As a further example, in one embodiment, "decoding" refers to entropy decoding only, in another embodiment, "decoding" refers to differential decoding only, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to the broader decoding process generally will be clear based on the context of a particular description and will be well understood by one of ordinary skill in the art.

[0126] Various implementations involve encoding. Similar to the above discussion regarding "decoding," "encoding," as used in this disclosure, can encompass, for example, all or some of the processes performed on an input video sequence to generate an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential encoding, transforming, quantizing, and entropy encoding. In various embodiments, such processes also, or alternatively, include processes performed by the encoders of various implementations described in this disclosure.

[0127] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to the broader encoding process in general will be clear based on the context of the particular description and will be well understood by one of ordinary skill in the art.

[0128] When a figure is presented as a flow diagram, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow diagram of the corresponding method / process.

[0129] Various embodiments refer to rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is typically considered, often given computational complexity constraints. Rate-distortion optimization is typically formulated to minimize a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, these approaches may be based on extensive testing of all encoding options, including all considered modes or coding parameter values, but with a thorough evaluation of their encoding costs and the associated distortion of the reconstructed signal after coding and decoding. In particular, faster approaches may be used to reduce encoding complexity by calculating approximate distortion based on a prediction or prediction residual signal rather than the reconstructed signal. These two approaches may also be used in combination, such as by using approximate distortion for only some of the possible encoding options and full distortion for others. Other approaches evaluate only a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform the optimization, but the optimization does not necessarily involve a thorough evaluation of both the encoding cost and the associated distortion.

[0130] Implementations and aspects described herein may be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed in the context of only one type of implementation (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., an apparatus or a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, a processor, which refers generally to processing devices including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

[0131] References to "one embodiment" or "one embodiment," or "one implementation" or "one implementation," as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with an embodiment is included in at least one embodiment. Thus, appearances of the phrases "in one embodiment," or "in one implementation," or "in one implementation," as well as any other variations, appearing in various places throughout this disclosure are not necessarily all referring to the same embodiment.

[0132] Additionally, this disclosure may refer to "determining" various portions of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0133] Additionally, this disclosure may refer to "accessing" various portions of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0134] Additionally, this disclosure may refer to "receiving" various portions of information. Receiving, like "accessing," is intended to be a broad term. Receiving information can include, for example, one or more of accessing information or retrieving information (e.g., from memory). Furthermore, "receiving" typically involves in some manner, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0135] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first listed option (A), or the selection of only the second listed option (B), or the selection of both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed option (A), or the selection of only the second listed option (B), or the selection of only the third listed option (C), or the selection of only the first and second listed options (A and B), or the selection of only the first and third listed options (A and C), or the selection of only the second and third listed options (B and C), or the selection of all three options (A, B, and C). This can be extended to a number of listed items.

[0136] Also, as used herein, the word "signaling" refers to, among other things, indicating something to a corresponding decoder. For example, in a particular embodiment, an encoder signals a specific one of multiple parameters for region-based filter parameter selection for artifact removal. In this way, in one embodiment, the same parameters are used on both the encoder and decoder sides. Thus, for example, an encoder can transmit a specific parameter to a decoder (explicit signaling), so that the decoder can use the same specific parameter. Conversely, if the decoder already has a specific parameter as well as other parameters, signaling can be used without transmission (implicit signaling), allowing the decoder to easily recognize and select the specific parameter. By avoiding the transmission of any actual function, bit savings are realized in various embodiments. It should be appreciated that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder in various embodiments. Although the above relates to the verb form of the word "signaling," the word "signaling" can also be used as a noun herein.

[0137] Implementations can generate various signals formatted to carry information that can be stored or transmitted, for example. The information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal can be formatted to carry a bitstream of the described embodiments. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0138] Several embodiments will be described, the features of which may be provided singly or in any combination across various claim categories and types.

[0139] It should be noted that various hardware elements of one or more of the described embodiments are referred to as "modules" that perform (i.e., perform, implement, etc.) various functions described herein with respect to the respective modules. As used herein, a module includes any hardware deemed suitable for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices). It should be noted that each described module may also include executable instructions to perform one or more functions described as being performed by the respective module, and these instructions may take the form of hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., or may be stored on any suitable non-transitory computer-readable medium(s), commonly referred to as RAM, ROM, etc.

[0140] Although features and elements are described above in particular combinations, those skilled in the art will understand that each feature or element can be used alone or in any combination with the other features and elements. Furthermore, the methods described herein may be implemented in a computer program, software, or firmware embodied in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random-access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, optical media such as CD-ROM disks, and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. 1. A method comprising: obtaining a container file including a plurality of haptic tracks, the container file including information associating each of the plurality of haptic tracks with at least one of a respective device, a respective perception, or a respective avatar; obtaining information indicative of a selection of at least one device, at least one perception, or at least one avatar; extracting haptic data in response to the selection, wherein the extracted haptic data excludes at least one of the plurality of haptic tracks that is not associated with any of the selected devices, senses, or avatars.

2. The method of claim 1 , wherein the method is performed by a server and the information indicating the selection is received from a client device.

3. The method of claim 1 or 2, further comprising providing the extracted haptic data to a client device in a bitstream.

4. The method of any one of claims 1 to 3, further comprising providing a manifest file indicating at least one available device, at least one available perception, or at least one available avatar, wherein the information indicating the selection is received in response to the manifest file.

5. The method of claim 1 or 2, further comprising providing the extracted haptic data to a client device as a container file.

6. The method of any one of claims 1 to 5, further comprising rendering the extracted haptic data.

7. The method of any one of claims 1 to 6, wherein the information associating a haptic track with a respective device includes a device identifier in a haptic channel configuration box associated with the respective track.

8. The method of any one of claims 1 to 7, wherein the information associating a haptic track with each perception includes information identifying a track group comprising a plurality of tracks associated with each perception.

9. The method of any one of claims 1 to 8, wherein the information associating a haptic track with a respective avatar comprises an avatar identifier in a haptic perception configuration box.

10. 10. The method of claim 1, wherein the container file includes information associating each of a plurality of the haptic tracks with a respective device, the selection being a selection of at least one device, and the extracted haptic data excludes at least one of the plurality of haptic tracks that is not associated with any selected device.

11. The method according to any one of claims 1 to 10, wherein the container file is an ISOBMFF file.

12. 1. An apparatus comprising one or more processors, the one or more processors performing at least: obtaining a container file including a plurality of haptic tracks, the container file including information associating each of the plurality of haptic tracks with at least one of a respective device, a respective perception, or a respective avatar; obtaining information indicative of a selection of at least one device, at least one perception, or at least one avatar; extracting haptic data in response to the selection, wherein the extracted haptic data excludes at least one of the plurality of haptic tracks that is not associated with any of the selected devices, senses, or avatars.

13. The apparatus of claim 12 , further configured to provide the extracted haptic data to a client device in a bitstream.

14. The apparatus of claim 12 , further comprising providing the extracted haptic data to a client device as a container file.

15. The apparatus of claim 12 , further comprising rendering the extracted haptic data.

Images