Carrying of encoded haptic data in media container

Abstract

Systems and methods for encoding, processing, and / or decoding container files (such as ISOBMFF container files) representing haptic data are described. A method according to some embodiments includes obtaining a container file including a plurality of haptic trajectories, the container file including information associating each of the plurality of haptic trajectories with at least one of a respective device, a respective perception, or a respective avatar; obtaining information indicative of a selection of at least one device, at least one perception, or at least one avatar; and extracting haptic data in response to the selection, wherein the extracted haptic data excludes at least one of the plurality of haptic trajectories that is not associated with any selected device, perception, or avatar.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a non-provisional application of U.S. Provisional Patent Application Serial No. 63 / 417,638, filed on October 19, 2022, entitled “Carriage of Coded Haptics Data in Media Containers,” and claims the benefit of 35 U.S.C. §119(e), which is incorporated herein by reference in its entirety. Technical Field

[0003] A haptic sequence is a set of encoded data rendered based on touch sensing and positioning in space, similar to how a video sequence is a set of encoded data rendered using visual sensing. A haptic sequence encodes temporal data, for example represented as a track associated with a haptic device. A haptic device can render different modalities of touch sensing and positioning in space, such as vibration, force, position, velocity, or temperature. Background Art

[0004] The Moving Picture Experts Group (MPEG) is currently developing a new standard (ISO / IEC 23090-31). Figure 2 The MPEG tactile codec architecture is shown. In this architecture, the encoded representation of tactile data can be in one of two formats: an interchange format (.hjif) or a distribution format (.hmpg). The interchange format is a human-readable description of the tactile data based on JSON, while the distribution format is a compressed binary representation of the data. The two formats have complementary purposes, and lossless or lossy one-to-one conversion can operate between them. The compressed binary bitstream in the distribution format is structured into a sequence of network abstraction layer (NAL) units to facilitate encapsulation by any network protocol or file format.

[0005] The haptic decoder takes as input a binary '.hmpg' file or a '.hjif' file and outputs a .hjif file. The haptic data contained in the resulting '.hjif' file can then be rendered directly on a haptic device, or using an intermediate synthesizer that generates pulse code modulation (PCM) data.

[0006] Figure 2 is a functional block diagram showing the tactile codec architecture corresponding to ISO / IEC 23090-31. The data structures of the two codec formats follow Figure 3 Hierarchical organization shown.

[0007] The top level of the structure describes the entire haptic experience defined in the file or stream. It contains some high-level metadata information. It also provides a list of avatars (body representations) that can be referenced to specify the desired location of haptic stimulation on the body. The haptic data itself is described by a list of "sensations". These perceptions correspond to haptic signals associated with specific sensory modalities (vibration, force, position, velocity, temperature, etc.).

[0008] In addition to perception-specific metadata, a perception contains a list of channels. Within a channel, the data is broken down into frequency bands. Each frequency band defines a portion of the signal within a given frequency range. Frequency bands are described using a list of haptic effects, each containing a list of keyframes. The haptic signal in a channel can then be reconstructed by combining data from different frequency bands (adding high and low frequency bands).

[0009] Figure 4 An example of a network abstraction layer (NAL) unit structure in a haptic bitstream is shown. Figure 5 Indicates the NAL unit payload type.

[0010] Within the ISO / IEC 14496 (MPEG-4) standard, there are several parts that define file formats for storing time-based media. These are based on and derived from the ISO Basic Media File Format (ISOBMFF), which is a structured, media-independent definition. ISOBMFF contains structured media data information that is primarily used to present media data such as audio, video, etc. in time. Non-time-based data is also supported, such as metadata at different levels within the file structure. The logical structure of the file is that of a "movie", which in turn contains a set of time-parallel "tracks". The temporal structure of the file is that the track contains a sequence of "samples" in time, and those sequences are mapped into the timeline of the entire movie. ISO BMFF is based on the concept of a box-structured file. A box-structured file consists of a series of boxes (sometimes called atoms) that have a size and a type. These types are 32-bit values ​​and are usually selected as four printable characters, also known as four-character codes (4CC). Non-scheduled data may be contained in a metadata box at the file level, or within a movie may be attached to one of the movie boxes or timed data streams (referred to as tracks).

[0011] At the top-level box within the ISOBMFF container is the MovieBox (‘moov’), which contains metadata for the continuous media streams present in the file. This metadata is signaled within the box hierarchy in the movie box (e.g., within a TrackBox (‘trak’)). A track represents a continuous media stream present in the file. The media stream itself consists of a sequence of samples, such as audio or video access units of a basic media stream, and is encapsulated within a MediaDataBox (‘mdat’) present at the top level of the container. The metadata for each track includes a list of sample description entries, each entry providing the encoding or encapsulation format for the track, as well as initialization data for processing that format. Each sample is associated with one of the sample description entries for the track. ISO / IEC 14496-12 provides tools for defining an explicit timeline graph for each track. This is referred to as an edit list and is signaled using an EditListBox with the following syntax, where each entry defines a part of the track timeline: by mapping a part of the synthetic timeline, or by indicating a “null” time (mapping to a part of the presentation timeline with no media content, a “null” edit).

[0012] Summary of the Invention

[0013] Systems and methods for encoding and / or decoding a container file (such as an ISOBMFF container file) representing haptic data are described. In an example, a haptic experience track is encoded in the container file. The haptic experience track includes information describing at least one available avatar for the haptic experience; and configuration information for at least one perception in the haptic experience. The haptic experience track may also reference at least one haptic track. The haptic track may include all frequency bands for a respective perception in the perception. The haptic track may have samples carrying haptic band data bitstream units.

[0014] Encoder and decoder devices are provided to perform the methods described herein. The encoder or decoder device may include a processor configured to perform the methods described herein. The device may include a computer-readable medium (e.g., a non-transitory medium) storing instructions for performing the methods described herein. In some embodiments, the computer-readable medium (e.g., a non-transitory medium) stores haptic data encoded using any of the methods described herein.

[0015] Methods according to some embodiments include: obtaining a container file, such as an ISOBMFF file including multiple tactile tracks, the container file including information associating each of the multiple tactile tracks with at least one of a corresponding device, a corresponding perception, or a corresponding avatar; obtaining information indicating a selection of at least one device, at least one perception, or at least one avatar; and extracting tactile data in response to the selection, wherein the extracted tactile data excludes at least one tactile track of the multiple tactile tracks that is not associated with any of the selected devices, perceptions, or avatars.

[0016] In some embodiments, the method is performed by a server and the information indicative of the selection is received from a client device.

[0017] Some embodiments further include providing the extracted haptic data in a bitstream to a client device.

[0018] Some embodiments further include providing a manifest file indicating at least one available device, at least one available perception, or at least one available avatar, wherein the information indicative of the selection is received in response to the manifest file.

[0019] Some embodiments further include providing the extracted haptic data to a client device as a container file.

[0020] Some embodiments further include rendering the extracted haptic data.

[0021] In some embodiments, the information associating the haptic track with the respective device includes a device identifier in a haptic channel configuration box associated with the respective track.

[0022] In some embodiments, the information associating the haptic tracks with the respective perceptions includes information identifying a track group including a plurality of tracks associated with the respective perceptions.

[0023] In some embodiments, the information associating the haptic track with the corresponding avatar includes an avatar identifier in a haptic perception configuration box.

[0024] In some embodiments, the container file includes information associating each of the plurality of haptic tracks with a respective device, the selection is a selection of at least one device, and the extracted haptic data excludes at least one haptic track of the plurality of haptic tracks that is not associated with any of the selected devices.

[0025] In some embodiments, a device includes one or more processors configured to at least perform: obtaining a container file, such as an ISOBMFF file including a plurality of haptic trajectories, the container file including information associating each of the plurality of haptic trajectories with at least one of a corresponding device, a corresponding perception, or a corresponding avatar; obtaining information indicating a selection of at least one device, at least one perception, or at least one avatar; and extracting haptic data in response to the selection, wherein the extracted haptic data excludes at least one haptic trajectory among the plurality of haptic trajectories that is not associated with any of the selected devices, perceptions, or avatars.

[0026] One or more of the present embodiments also provide a computer-readable storage medium having stored thereon instructions for performing any of the methods described herein. Some embodiments include a computer-readable storage medium having stored thereon a bitstream or a container file generated according to the methods described herein. Some embodiments include a computer program product including instructions for performing any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1A is a system diagram showing an example communication system in which one or more of the disclosed embodiments may be implemented.

[0028] Figure 1B is a system diagram showing an example wireless transmit / receive unit (WTRU) that may be used within the communication system shown in Figure 1A and shown in.

[0029] Figure 1C is a functional block diagram of a system used in some embodiments described herein.

[0030] Figure 2 shows an MPEG haptic codec architecture.

[0031] Figure 3 shows a hierarchical data structure of two codec formats.

[0032] Figure 4 shows the NAL unit structure in a haptic bitstream.

[0033] Figure 5 shows the NAL unit payload type.

[0034] Figure 6 shows a method performed by a server in some embodiments.

[0035] Figure 7 shows a method performed by a client in some embodiments.

[0036] Example Network for Implementation

[0037] Figure 1A 1 is a diagram illustrating an example communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content such as voice, data, video, messaging, broadcast, etc. to multiple wireless users. The communication system 100 may enable multiple wireless users to access such content by sharing system resources (including wireless bandwidth). For example, the communication system 100 may use one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single carrier FDMA (SC-FDMA), zero tail unique word DFT-spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block filtered OFDM, and filter bank multi-carrier (FBMC), etc.

[0038] like Figure 1A As shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, a RAN 104, a CN 106, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, but it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. For example, the WTRUs 102a, 102b, 102c, 102d (any of which may be referred to as a "station" and / or "STA") may be configured to send and / or receive wireless signals and may include user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular phone, a personal digital assistant (PDA), a smart phone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (IoT) device, a watch or other wearable device, a head-mounted display (HMD), a vehicle, a drone, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in the context of an industrial and / or automated process chain), consumer electronic devices, devices operating on a commercial and / or industrial wireless network, etc. Any of the WTRUs 102a, 102b, 102c, and 102d may be interchangeably referred to as a UE.

[0039] The communication system 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to facilitate access to one or more communication networks (e.g., the CN 106, the Internet 110, and / or other networks 112) of the WTRUs 102a, 102b, 102c, 102d by connecting to at least one wireless interface. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node B, an eNode B, a Home Node B, a Home eNode B, a gNB, an NR Node B, a site controller, an access point (AP), a wireless router, and the like. Although each of the base stations 114a, 114b is depicted as a single component, it will be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0040] The base station 114a may be part of the RAN 104, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), a relay node, etc. The base station 114a and / or the base station 114b may be configured to send and / or receive wireless signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be in a licensed spectrum, an unlicensed spectrum, or a combination of a licensed spectrum and an unlicensed spectrum. A cell may provide coverage for wireless services to a specific geographic area, which may be relatively fixed or may change over time. The cell may be further divided into cell sectors. For example, a cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, that is, each transceiver corresponds to a sector of the cell. In an embodiment, the base station 114a may use multiple-input multiple-output (MIMO) technology, and may use multiple transceivers for each sector of the cell. For example, beamforming may be used to send and / or receive signals in a desired spatial direction.

[0041] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0042] More specifically, as described above, the communication system 100 may be a multiple access system and may use one or more channel access schemes such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, etc. For example, the base station 114a in the RAN 104 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may use Wideband CDMA (WCDMA) to establish the air interface 116. WCDMA may include communication protocols such as High Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High Speed ​​UL Packet Access (HSUPA).

[0043] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro).

[0044] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as NR radio access, which may establish the air interface 116 using New Radio (NR).

[0045] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may together implement LTE radio access and NR radio access, for example using dual connectivity (DC) principles. Thus, the air interface used by the WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies, and / or transmissions to / from multiple types of base stations (e.g., eNBs and gNBs).

[0046] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi)), IEEE 802.16 (Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile Communications (GSM), GSM Enhanced Data Rates for Evolution (EDGE), GSM EDGE (GERAN), etc.

[0047] As an example, Figure 1A The base station 114b in the example may be a wireless router, a Home NodeB, a Home eNodeB, or an access point, and may use any appropriate RAT to facilitate wireless connectivity in a local area, such as a business location, a residence, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a road, and the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d may establish a wireless local area network (WLAN) by implementing a radio technology such as IEEE 802.11. In an embodiment, the base station 114b and the WTRUs 102c, 102d may establish a wireless personal area network (WPAN) by implementing a radio technology such as IEEE 802.15. In another embodiment, the base station 114b and the WTRUs 102c, 102d may establish a picocell or a femtocell by using a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-APro, NR, etc.). Figure 1A As shown, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not be required to access the Internet 110 via the CN 106.

[0048] The RAN 104 may be in communication with the CN 106, which may be any type of network configured to provide voice, data, applications, and / or Voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. Data may have varying quality of service (QoS) requirements, such as different throughput requirements, latency requirements, fault tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, etc. The CN 106 may provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, etc., and / or perform high-level security functions such as user authentication. Although in Figure 1AAlthough not shown, it will be appreciated that the RAN 104 and / or the CN 106 may be in direct or indirect communication with other RANs that employ the same RAT or a different RAT as the RAN 104. For example, in addition to being connected to the RAN 104, which may employ NR radio technology, the CN 106 may also be in communication with other RANs (not shown) employing GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0049] The CN 106 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network that provides plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the Transmission Control Protocol (TCP), the User Datagram Protocol (UDP), and / or the Internet Protocol (IP) in the TCP / IP Internet protocol suite. The networks 112 may include wired and / or wireless communication networks owned and / or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs, which may use the same RAT as the RAN 104 or a different RAT.

[0050] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communication system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). Figure 1A The illustrated WTRU 102c may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[0051] Figure 1B is a system diagram showing an example WTRU 102. Figure 1B As shown, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keyboard 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138. It will be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0052] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. Although Figure 1B The processor 118 and the transceiver 120 are depicted as separate components, but it will be understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0053] The send / receive element 122 may be configured to send or receive signals to or from a base station (e.g., base station 114a) via an air interface 116. For example, in one embodiment, the send / receive element 122 may be an antenna configured to send and / or receive RF signals. In an embodiment, the send / receive element 122 may be a transmitter / detector configured to send and / or receive, for example, IR, UV, or visible light signals. In yet another embodiment, the send / receive element 122 may be configured to send and / or receive both RF signals and optical signals. It will be appreciated that the send / receive element 122 may be configured to send and / or receive any combination of wireless signals.

[0054] Although the transmit / receive element 122 is Figure 1B 1 as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0055] The transceiver 120 may be configured to modulate signals to be transmitted by the transmit / receive element 122 and to demodulate signals received by the transmit / receive element 122. As described above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0056] The processor 118 of the WTRU 102 may be coupled to a speaker / microphone 124, a keyboard 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit), and may receive user input data from these components. The processor 118 may also output user data to the speaker / microphone 124, the keyboard 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from and store data in any type of suitable memory, such as a non-removable memory 130 and / or a removable memory 132. The non-removable memory 130 may include a random access memory (RAM), a read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, etc. In other embodiments, the processor 118 may access information from and store data in a memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[0057] The processor 118 may receive power from the power source 134, and may be configured to distribute and / or control power for use by the other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0058] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to or in lieu of the information from the GPS chipset 136, the WTRU 102 may receive location information from a base station (e.g., base stations 114a, 114b) over the air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information via any suitable location-determination method while remaining consistent with the embodiments.

[0059] The processor 118 may also be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for taking pictures and / or videos), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, module, a frequency modulation (FM) radio unit, a digital music player, a media player, a video game console module, an Internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, etc. The peripheral device 138 may include one or more sensors, which may be one or more of a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor, a geolocation sensor, an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0060] The WTRU 102 may include a full-duplex radio for which some or all signals may be transmitted and received (e.g., transmit and receive). For example, subframes associated with specific subframes for both UL (e.g., for transmission) and downlink (e.g., for reception) may be concurrent and / or simultaneous. The full-duplex radio may include an interference management unit to reduce and / or substantially eliminate self-interference via hardware (e.g., choke) or via signal processing by a processor (e.g., a separate processor (not shown) or via processor 118). In an embodiment, the WRTU 102 may include a half-duplex radio for which some or all signals may be transmitted and received (e.g., associated with specific subframes for UL (e.g., for transmission) or downlink (e.g., for reception).

[0061] Although the WTRU Figure 1A and Figure 1B Although described as a wireless terminal, it is contemplated that in certain representative embodiments, such a terminal may use (eg, temporarily or permanently) a wired communication interface with a communication network.

[0062] In a representative embodiment, other network 112 may be a WLAN.

[0063] based on Figure 1A and Figure 1B As well as the corresponding description, one or more or all of the functions described herein may be performed by one or more simulation devices (not shown). A simulation device may be one or more devices configured to simulate one or more or all of the functions described herein. For example, a simulation device may be used to test other devices and / or simulate network and / or WTRU functions.

[0064] The simulation device can be designed to implement one or more tests of other devices in a laboratory environment and / or an operator network environment. For example, one or more simulation devices can perform one or more or all functions when fully or partially implemented and / or deployed as part of a wired and / or wireless communication network in order to test other devices within the communication network. One or more simulation devices can perform one or more or all functions when temporarily implemented / deployed as part of a wired and / or wireless communication network. For testing purposes, the simulation device can be directly coupled to another device, and / or over-the-air wireless communication can be used to perform testing.

[0065] One or more simulation devices can perform one or more functions, including all functions, without being implemented / deployed as part of a wired and / or wireless communication network. For example, the simulation device can be used in a test scenario in a test lab and / or a non-deployed (e.g., testing) wired and / or wireless communication network to enable testing of one or more components. One or more simulation devices can be test devices. Direct RF coupling and / or wireless communication via RF circuits (e.g., which can include one or more antennas) can be used by the simulation device to send and / or receive data.

[0066] Example System

[0067] The embodiments described herein are not limited to implementation on a WTRU. Such embodiments may be implemented using other systems, such as Figure 1C system. Figure 1C It is a block diagram of an example of a system that implements various aspects and embodiments. System 1000 can be implemented as a device including various components described below, and is configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances and servers. The elements of system 1000 can be implemented as a single integrated circuit (IC), multiple ICs and / or discrete components individually or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed on multiple ICs and / or discrete components. In various embodiments, system 1000 is coupled to one or more other systems or other electronic devices via, for example, a communication bus or by a dedicated input and / or output port. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0068] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein to implement, for example, the various aspects described in this document. The processor 1010 may include embedded memory, input and output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include a non-volatile memory and / or a volatile memory, including but not limited to an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a programmable read-only memory (PROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a magnetic disk drive, and / or an optical disk drive. As a non-limiting example, the storage device 1040 may include an internal storage device, an attached storage device (including a removable and non-removable storage device), and / or a network accessible storage device.

[0069] The system 1000 includes an encoder / decoder module 1030, which is configured to process data, for example, to provide encoded video or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 1030 may be implemented as a separate element in the system 1000, or may be incorporated into the processor 1010 as a combination of hardware and software, as known to those skilled in the art.

[0070] Program code to be loaded onto the processor 1010 or the encoder / decoder 1030 to perform various aspects described in this document may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of the items during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operation logic.

[0071] In some embodiments, memory internal to the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store, for example, the operating system of the television. In at least one embodiment, fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by JVET (Joint Video Experts Group)).

[0072] As shown in block 1130, input to the elements of system 1000 may be provided through various input devices. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives RF signals transmitted, for example, by a broadcaster over the air, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 1C Other examples not shown include composite video.

[0073] In various embodiments, the input device of block 1130 has associated corresponding input processing elements as known in the art. For example, the RF portion may be associated with elements suitable for performing the following operations: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) band-limiting the frequency band to a narrower frequency band to select a signal frequency band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data stream packet. The RF portion of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a frequency band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF portion may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or down-converting to baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (for example cable) medium, and filter, down-convert and filter to the desired frequency band to perform frequency selection. Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements of similar or different functions. Adding element can include inserting element between existing element, for example inserting amplifier and analog-to-digital converter. In various embodiments, the RF part comprises antenna.

[0074] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It will be appreciated that various aspects of input processing, such as Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or in the processor 1010 as desired. Similarly, aspects of USB or HDMI interface processing may be implemented, for example, in a separate interface IC or in the processor 1010 as desired. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 1010 and the encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on an output device.

[0075] The various elements of system 1000 may be disposed within an integrated housing in which the various elements may be interconnected and transmit data between them using an appropriate connection arrangement 1140, such as an internal bus known in the art, including an inter-IC (I2C) bus, wiring, and printed circuit boards.

[0076] The system 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data through the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0077] In various embodiments, data is streamed or otherwise provided to the system 1000 using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signals of these embodiments are received through a communication channel 1060 and a communication interface 1050 adapted for Wi-Fi communications. The communication channel 1060 of these embodiments is typically connected to an access point or router that provides access to an external network including the Internet to allow streaming applications and other over-the-top communications. Other embodiments provide streaming data to the system 1000 using a set-top box that delivers data through an HDMI connection of an input box 1130. Other embodiments provide streaming data to the system 1000 using an RF connection of an input box 1130. As described above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use a wireless network other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0078] The system 1000 can provide output signals to various output devices, including a display 1100, a speaker 1110, and other peripherals 1120. The display 1100 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1100 can be used for a television, a tablet computer, a laptop computer, a mobile phone (mobile phone), or other devices. The display 1100 can also be integrated with other components (e.g., as in a smart phone), or be separate (e.g., an external monitor for a laptop computer). In various examples of embodiments, other peripherals 1120 include one or more of a separate digital video disk (or digital versatile disk) (DVR, for both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripherals 1120 that provide functions based on the output of the system 1000. For example, a disk player performs the function of playing the output of the system 1000.

[0079] In various embodiments, control signals are communicated between the system 1000 and the display 1100, speaker 1110, or other peripheral device 1120 using signaling such as AV.Link, consumer electronics control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices may be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, output devices may be connected to the system 1000 using a communication channel 1060 via a communication interface 1050. In an electronic device such as, for example, a television, the display 1100 and speaker 1110 may be integrated into a single unit with other components of the system 1000. In various embodiments, the display interface 1070 includes a display driver, such as, for example, a timing controller (TCON) chip.

[0080] For example, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 may alternatively be separate from one or more other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0081] The embodiments may be implemented by computer software implemented by the processor 1010, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. The processor 1010 may be of any type suitable for the technical environment and may encompass one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture as non-limiting examples. DETAILED DESCRIPTION

[0082] Problems Solved in Some Embodiments

[0083] While the bitstream format developed in ISO / IEC 23090-31 enables the description of haptic experiences in a compact representation that can be easily processed by a haptic decoder, the bitstream format does not allow scalable access to different stream components and does not enable selective streaming. Furthermore, an immersive experience involves multiple different media types that are synchronized during playback to provide a target experience. There is currently no well-defined method for storing and multiplexing such a haptic bitstream with other types of media (such as audio and video) in an ISOBMFF media container.

[0084] This disclosure describes a generic and scalable design to support carrying a haptic bitstream generated by an ISO / IEC 23090-31 haptic codec in an ISOBMFF container. Example embodiments can be applied to ecosystems involving immersive media encoding, storage and streaming of encoded haptic media content, and decoding of haptic media data on devices, or any service that provides an immersive media experience.

[0085] Tactile experience track

[0086] In an example embodiment, the main entry for a haptic experience is a haptic experience track. A haptic experience track may have a sample entry of type HapticsSampleEntry, with a 'hexp' four-character code (4CC). A HapticsSampleEntry with a 'hexp' 4CC contains a HapticsExperienceConfigurationBox that describes the configuration of the haptic experience, including a description of the available avatars in the experience and the configuration of the various perceptions therein. A haptic experience track may reference one or more haptic tracks. Each haptic track may carry frequency band data for a specific channel for a specific perception in the haptic experience.

[0087] To link a haptic experience track with an associated haptic track, the track reference facility of ISO / IEC 14496-12 may be used. A TrackReferenceTypeBox with reference type 'hpbd' may be added to the TrackReferenceBox within the track box of the haptic experience track. The TrackReferenceTypeBox may contain an array of track_ids specifying identifiers for referenced haptic tracks.

[0088] In another embodiment, each haptic trace carries all frequency bands belonging to a specific channel of one of the perceptions of the haptic experience.

[0089] aligned(8)class HapticsSampleEntry('hexp')extends SampleEntry{

[0090] HapticsExperienceConfigurationBox();

[0091]

[0092] In the example, the semantics of the fields of the HapticsExperienceConfigurationBox are as follows:

[0093] creation_time is an integer declaring the creation time of this haptic experience (in seconds since midnight January 1, 1904, UTC).

[0094] description is a human-readable description of the experience.

[0095] In the example, the semantics of the HapticsAvatarDescriptionBox field are as follows:

[0096] avatar_count is the number of avatars available in the haptic experience.

[0097] avatar_id is a unique identifier for the avatar.

[0098] · level_of_detail indicates the level of detail of the avatar. Depending on the application, the 3D mesh corresponding to the avatar provides high resolution and accuracy with variable vertex density. Therefore, a certain mesh resolution can be related to the tactile spatial acuity of the associated tactile perception.

[0099] avatar_type indicates the type of tactile perception represented by the avatar based on the spatial acuity of the corresponding tactile modality. Examples of possible values ​​are given in Table 1.

[0100] Table 1 Avatar types

[0101]

[0102]

[0103] mesh_uri provides the URI to access the associated custom mesh file for the avatar. Only present if the value of avatar_type is 0.

[0104] avatar_type specifies the type of tactile sensation represented by the avatar. Values ​​can be vibration, pressure, temperature, or custom. For a "custom" mesh, the mesh is provided as a companion file. If the type is one of the first three, the avatar mesh may be predefined for applications targeting a specific modality. If you have multiple modalities, a custom avatar will probably be the best option.

[0105] In the example, HapticSpectionConfigurationBox has the following syntax:

[0106]

[0107]

[0108] In the example, the semantics of the HapticSpectionConfigurationBox fields are as follows.

[0109] perception_count is the number of perceptions in the tactile experience.

[0110] perception_id is a unique identifier for a perception.

[0111] description is a human-readable description of the perception.

[0112] modality indicates the type of perception. Examples of possible values ​​are given in Table 2.

[0113] value describe 0 other 1 pressure 2 Acceleration 3 speed 4 Location 5 temperature 6 Vibrotactile 7 water 8 wind 9 force 10 Vibration tactile texture 11 Stiffness 12 friction

[0114] Table 2. Perception modality

[0115] • avatar_id is the avatar identifier for the avatar body model associated with the perception.

[0116] unit_exponent is the 10^x exponent that identifies the SI unit in the representation space of the argument.

[0117] perception_unit_exponent refers to the 10^x exponent of the SI unit measure of the dependent variable.

[0118] channels_count is the number of tactile channels for sensing.

[0119] devices_count is the number of haptic reference devices associated with the perception.

[0120] device_id Unique identifier of the device.

[0121] device_type indicates the type of actuator. Possible values ​​are given in Table 3.

[0122] Table 3. Types of reference haptic devices

[0123] value describe 0 unknown 1 Linear Resonant Actuator 2 Voice Coil Actuator (VCA) 3 Eccentric Rotating Mass (ERM) 4 Piezoelectric

[0124] body_part_mask is a binary mask that specifies the location of the device or actuator on the body.

[0125] The semantics of each bit in the bit mask in the example can be described in the following table.

[0126]

[0127]

[0128] · device_mask is the bit mask associated with the device.

[0129] maximum_frequency indicates the maximum frequency of the actuator in Hertz (Hz).

[0130] minimum_frequency indicates the maximum frequency of the actuator in Hertz (Hz).

[0131] resonance_frequency indicates the resonant frequency of the actuator in Hertz (Hz).

[0132] maximum_amplitude indicates the maximum amplitude value of the target device according to the sensing modality.

[0133] impedance indicates the impedance of the actuator in ohms (Ω).

[0134] maximum_voltage indicates the maximum voltage of the actuator in volts (V).

[0135] maximum_current indicates the maximum current of the actuator in amperes (A).

[0136] maximum_displacement indicates the maximum displacement of the actuator in millimeters (mm).

[0137] ·weight indicates the weight of the device in kilograms (Kg).

[0138] ·size indicates the size of the device in millimeters (mm).

[0139] user_defined_data can be used to specify additional properties for the target device.

[0140] In another embodiment, the HapticSpeception data structure is defined as follows, where it contains another dedicated ISOBMFF structure for signaling information about the reference device.

[0141]

[0142] In the example, HapticsReferenceDevicesBox can be defined as follows.

[0143]

[0144] In the example, the semantics of the fields of HapticsReferenceDevicesBox can be as follows:

[0145] devices_count is the number of haptic reference devices associated with the perception.

[0146] In some embodiments, HapticseEffectsLibraryBox contains an optional box that can be present when an effects library is present in the haptic perception, and the band data unit can reference effects in the library without carrying the definition of the effects in the band data.HapticseEffectsLibraryBox can be constructed as follows.

[0147]

[0148]

[0149] In the example, the semantics of the fields of the HapticsEffectSlibraryBox are as follows.

[0150] effects_count is the number of effects in the effects library.

[0151] effect is an instance of HapticsLibraryEffect.

[0152] • effect_id indicates a unique identifier for the library effect.

[0153] effect_type indicates the type of effect. Examples of allowed values ​​are shown in the following table.

[0154] value describe 0 Base 1 refer to 2 Timeline

[0155] Table 4. Haptic effect types

[0156] position indicates the time position of the effect relative to the start of the experience.

[0157] Phase indicates the phase of the effect.

[0158] · band_type indicates the type of data in the band. Examples of possible values ​​are given in Table 5.

[0159] value describe 0 Instant 1 curve 2 Vector Wave 3 Not allowed

[0160] Table 5. Frequency band types

[0161] curve_type indicates the interpolation function for a band of type "curve". It is only present if the value of band_type is 1. Examples of possible values ​​are given in Table 6.

[0162] value describe 0 unknown 1 cube 2 Linear 3 Akima 4 Bessel 5 B-spline

[0163] Table 6. Curve band types

[0164] base_signal_type indicates the kind of base signal. Examples of possible values ​​are given in Table 7.

[0165] value describe 0 Sine 1 square 2 Triangle 3 Sawtooth Upward 4 Sawtooth Downward

[0166] Table 7. Basic signal types

[0167] • timeline_effects_count indicates the number of effects available in the timeline for this band.

[0168] keyframe_id is a unique identifier for a keyframe for the effect.

[0169] amplitude is the amplitude of the keyframe.

[0170] ·position is the relative position of the keyframe.

[0171] frequency is the relative frequency of the keyframe.

[0172] In another embodiment, the HapticsEffectsLibraryBox may be defined as follows.

[0173]

[0174] The semantics of the fields of the HapticsEffectSlibraryBox in this embodiment may be as follows.

[0175] • band_data_unit is an example of a bitstream data unit carrying band data.

[0176] Tactile Tracks

[0177] According to some embodiments, a haptic track is provided as a track whose samples carry haptic band data bitstream units. Similar to a haptic experience track, a haptic track also contains HapticsSampleEntry, but with a different 4CC type. To distinguish between a haptic track with samples carrying data of a single band that conveys a perception and channel combination and a haptic track where samples carry data of all bands of a certain perception channel, two different 4CCs can be used. For example, the 'hpd1' 4CC can be used for the former, while the 'hpd2' 4CC can be used for the latter.

[0178] In some embodiments, the definition of a HapticsSampleEntry of the 'hpd1' type is as follows:

[0179]

[0180] In some embodiments, the definition of a HapticsChannelConfigurationBox is as follows.

[0181]

[0182]

[0183] The corresponding semantics of the fields of the HapticsChannelConfigurationBox can be as follows:

[0184] · channel_id is the unique identifier for the channel.

[0185] · direction_present_flag is a flag indicating whether a direction is associated with the channel.

[0186] · device_id is the identifier for the reference device associated with the channel.

[0187] · gain indicates the gain associated with the channel to adapt the normalized encoded data value to a typical device.

[0188] · mixing_weight indicates the weight of the channel when different channels are mixed together to produce the final signal.

[0189] · body_part_mask is a bitmask indicating the location of the effect on the body.

[0190] · frequency_sampling indicates the sampling frequency of the original encoded signal in Hertz (Hz).

[0191] samples_count indicates the number of samples of the original coded signal. This field is present when the frequency_sampling value is greater than 0.

[0192] bands_count is the number of frequency bands available for this channel.

[0193] direction_x indicates the horizontal direction to the left / right in local space.

[0194] direction_y indicates the vertical direction up / down in local space.

[0195] direction_z indicates the forward / backward direction in local space.

[0196] vertices_count is the number of avatar vertices affected by the effect.

[0197] In some embodiments, the definition of HapticsBandConfigurationBox is as follows: classHapticsBandConfigurationBox extends FullBox('hbdC', version=0, flags=0) {

[0198] unsigned int(32)band_id;

[0199] unsigned int(2)band_type;

[0200] if(band_type == 1){

[0201] unsigned int(3)curve_type;

[0202] }

[0203] if((band_type==2)||(band_type==3)){

[0204] unsigned int(3)window_length;

[0205] }

[0206] unsigned int(16)freq_low;

[0207] unsigned int(16)freq_high;

[0208] }

[0209] In an example, the semantics of the fields of HapticsBandConfigurationBox may be as follows.

[0210] • band_id is a unique identifier for a frequency band.

[0211] · band_type indicates the type of frequency band. Examples of possible values ​​are given in Table 8.

[0212] value describe 0 Instant 1 curve 2 Vector Wave 3 Xiaobobo

[0213] Table 8. Frequency band types

[0214] • curve_type indicates the interpolation function used when band_type is value 1 (ie, curve band).

[0215] · window_length is the duration of the haptic keyframe.

[0216] freq_low is the lower frequency limit of the frequency band.

[0217] freq_high is the upper frequency limit of the band.

[0218] Haptic Track Sample

[0219] Depending on the sample entry type as defined by the 4CC of the sample entry, for a haptic track, the samples of the track may carry data for only one frequency band of a perceptual channel, or data for all frequency bands of a perceptual channel. When all frequency bands are carried in a sample of a haptic track, each sample may consist of multiple subsamples, where each subsample contains data for one of the frequency bands.

[0220] In another embodiment, keyframes for haptic effects for a channel band can be stored in separate samples in the haptic track, where each sample is assigned a decoding timestamp and a composition timestamp corresponding to the relative position of the keyframe relative to the start of the haptic experience. To identify samples belonging to a certain effect, multiple sample groups can be defined in the metadata of the ISOBMFF track. Each sample containing data for an independent keyframe (e.g., the first keyframe of an effect) can be designated as a synchronization sample to enable random access in the track.

[0221] Grouping tactile trajectories for tactile perception

[0222] When multiple haptic tracks are used to carry frequency band data for each channel of a haptic experience perception, in some embodiments, track grouping can be used to identify which haptic tracks are associated with a certain haptic perception.

[0223] In some embodiments, this can be accomplished by defining a track group type that extends the TrackGroupTypeBox defined in ISO / IEC 14496-12, which contains a track_group_id representing an identifier for a track group, and a track_group_type field storing a four-character code identifying the group type. The track_group_id and track_group_type pair identifies a track group within a container file.

[0224] In some embodiments, an example HapticsTrackGroupBox may be defined as follows.

[0225] Frame type: 'hptg'

[0226] Container: TrackGroupBox

[0227] Mandatory: None

[0228] Quantity: zero or more

[0229] aligned(8)class HapticsTrackGroupBox extends TrackGroupTypeBox('hptg'){

[0230] / / Here you can define additional data related to tactile perception

[0231] }

[0232] Example methods and systems

[0233] By using the systems, methods, and data structures as described herein, a player can extract only the bitstream elements it needs, such as only those bitstream elements related to a particular avatar, or those belonging to a particular perception. This may be particularly useful in the context of streaming. In the case of streaming, the exposure may be included in the manifest file.

[0234] Compared to systems that use only a single track to carry the haptic bitstream, example embodiments allow for a multi-track design where data pertaining to certain channels and / or senses is carried in its own track. In some embodiments, all frequency band data for a certain effect is provided in one sample. Some embodiments allow for a parser to extract certain frequency bands from a sample by using subsamples. The systems and methods described herein allow for container files to be constructed in a way that makes it easier for a player to extract only the necessary data.

[0235] In such Figure 6In the example method shown in the flowchart of , the method performed in some embodiments includes: at 602, obtaining a container file including a plurality of tactile tracks, the container file including information associating each of the plurality of tactile tracks with at least one of a corresponding device, a corresponding perception, or a corresponding avatar. At 604, obtaining information indicating a selection of at least one device, at least one perception, or at least one avatar. At 606, tactile data is extracted in response to the selection, wherein the extraction is performed so as to exclude at least one tactile track of the plurality of tactile tracks that is not associated with any of the selected devices, perceptions, or avatars. For example, the extracted tactile data may include all tracks associated with the selected at least one device, perception, or avatar, and in some embodiments, all remaining tactile tracks are excluded from the extracted data. In some embodiments, the extracted tactile data may include all tracks associated with the selected device, the selected perception, and the selected avatar, and in some embodiments, all remaining tactile tracks are excluded from the extracted data.

[0236] In some embodiments, Figure 6 The method may be performed by a server, and the selection information received at 604 may be received from a client device. In some such embodiments, the extracted tactile data may be provided to the client device in a bitstream. In some such embodiments, the server optionally provides a manifest file indicating at least one device, at least one available perception, or at least one available avatar at 610, wherein the information indicating the selection is received in response to the manifest file.

[0237] As an example, a server may obtain a container file having a first plurality of tactile tracks associated with a first perception, and a second plurality of tactile tracks associated with a second perception. The first perception may include information indicating that the modality of the first perception is "wind". The second perception may include information indicating that the modality of the second perception is "vibratory tactile". The server may use metadata in the container file to provide a manifest file (e.g., at 610) having information characterizing the first perception and the second perception. In response to the manifest file, the client may request only the second perception (vibratory tactile) without requesting the first perception (wind). For example, the client may not have any device capable of rendering the "wind" feeling, or the user may prefer not to experience simulated wind. In response to the client selection, the tactile data provided to the client excludes data based on tracks associated with the first perception. Similar selections may be made based on available avatars and / or available devices. For example, the client may not have some types of tactile devices, so only tactile data associated with the device that the client does have can be streamed to the client.

[0238] In such Figure 7In the example method shown in the flowchart of , the method performed in some embodiments includes obtaining a container file including a plurality of tactile tracks at 702, the container file including information associating each of the plurality of tactile tracks with at least one of a corresponding device, a corresponding perception, or a corresponding avatar. At 704, information indicating a selection of at least one device, at least one perception, or at least one avatar is obtained. The information may include configuration information about the type of tactile devices available to the user and / or the type of tactile experience that the user likes or would tend to avoid. At 706, tactile data is extracted in response to the selection, wherein the extraction is performed so as to exclude at least one tactile track of the plurality of tactile tracks that is not associated with any of the selected devices, perceptions, or avatars. At 708, the client may render the extracted tactile data using an appropriate actuator.

[0239] Further Embodiments

[0240] A method according to some embodiments includes encoding a haptic experience track in a container file, wherein the haptic experience track includes: information describing at least one avatar available for the haptic experience; and configuration information for at least one perception in the haptic experience.

[0241] A method according to some embodiments includes decoding a haptic experience track from a container file, wherein the haptic experience track includes: information describing at least one available avatar for the haptic experience; and configuration information for at least one perception in the haptic experience.

[0242] In some embodiments, the haptic experience track also references at least one haptic track.For example, the haptic experience track can include an array of identifiers of corresponding reference haptic tracks.

[0243] In some embodiments, each of the haptic traces includes all frequency bands of the channels for a corresponding one of the perceptions.

[0244] In some embodiments, the haptic trace includes frequency band data for at least one sensed channel.

[0245] In some embodiments, a haptic track is a track having samples that carry haptic frequency band data bitstream units.

[0246] In some embodiments, a haptic trace includes a code indicating whether (i) the haptic trace includes samples carrying data for a single frequency band for a sensory and channel combination, or (ii) the haptic trace includes samples carrying data for all frequency bands for a sensory channel.

[0247] In some embodiments, samples of a haptic trace contain data for only one frequency band of a sensory channel.

[0248] In some embodiments, a sample of the haptic trace contains data for all frequency bands of the sensory channel. In some such embodiments, each of the samples includes a plurality of sub-samples, and each of the sub-samples includes data for one of the frequency bands.

[0249] In some embodiments, the plurality of samples are key frame samples.

[0250] In some embodiments, a plurality of key frame samples are designated as synchronization samples.

[0251] In some embodiments, the file contains metadata defining at least one sample group, samples in the common sample group being samples belonging to a specified effect.

[0252] In some embodiments, the file also includes haptic track group information associating the plurality of haptic tracks with corresponding haptic track groups.

[0253] The present disclosure describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are specifically described, and at least in order to illustrate a single feature, are usually described in a manner that may appear restrictive. However, this is for the purpose of describing clearly, and does not limit the scope of the present disclosure or those aspects. In fact, all different aspects can be combined and interchanged to provide additional aspects. In addition, these aspects can also be combined and interchanged with the aspects described in the earlier application.

[0254] The aspects described and contemplated in this disclosure may be implemented in many different forms. Although some embodiments are specifically shown, other embodiments are contemplated, and the discussion of specific embodiments does not limit the breadth of implementation. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to sending a generated or encoded bitstream. These and other aspects may be implemented as methods, devices, computer-readable storage media having stored thereon instructions for encoding or decoding video data according to any of the methods described, and / or computer-readable storage media having stored thereon a bitstream generated according to any of the methods described.

[0255] In this disclosure, the terms "reconstruction" and "decoding" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image", "picture" and "frame" may be used interchangeably. Typically, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0256] Various methods are described herein, and each method includes one or more steps or actions for realizing the described method.Unless the correct operation of the method requires the steps or actions of a specific order, otherwise the order and / or use of specific steps and / or actions can be modified or combined.In addition, terms such as "first", "second" etc. can be used to modify elements, components, steps, operations, etc. in various embodiments, such as, for example, "first decoding" and "second decoding".Unless specifically needed, the use of these terms does not mean the sequencing of modified operations.Therefore, in this example, the first decoding does not need to be performed before the second decoding, and can, for example, occur before, during, or in a time period overlapping with the second decoding.

[0257] For example, various numerical values ​​may be used in the present disclosure. The specific values ​​are for example purposes, and the described aspects are not limited to these specific values.

[0258] The embodiments described herein may be implemented by computer software or other hardware implemented by a processor, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The processor may be of any type suitable for the technical environment, and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture as non-limiting examples.

[0259] Various implementations involve decoding. "Decoding" as used in the present disclosure may encompass, for example, all or part of the processing performed on a coded sequence received by an interface connection in order to produce a final output suitable for display. In various embodiments, such processing includes one or more of the processing typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processing also or alternatively includes processing performed by a decoder of the various implementations described in the present disclosure, such as extracting a picture from a spliced ​​(packed) picture, determining the upsampling filter to be used and then upsampling the picture, and flipping the picture back to its intended orientation.

[0260] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Based on the context of the particular description, it will be clear whether the phrase "decoding process" is intended to refer specifically to a subset of operations or generally to a broader decoding process.

[0261] Various implementations involve encoding. In a manner similar to the above discussion of "decoding", "encoding" as used in the present disclosure may encompass, for example, all or part of the processing performed on an input video sequence in order to produce an encoded bitstream. In various embodiments, such processing includes one or more processes typically performed by an encoder, such as partitioning, differential encoding, transforms, quantization, and entropy encoding. In various embodiments, such processing also or alternatively includes processing performed by an encoder of the various implementations described in the present disclosure.

[0262] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Based on the context of the particular description, it will be clear whether the phrase "encoding process" is intended to refer specifically to a subset of operations or generally to a broader encoding process.

[0263] When the figures are presented as flow charts, it should be understood that they also provide block diagrams of the corresponding apparatus. Similarly, when the figures are presented as block diagrams, it should be understood that they also provide flow charts of the corresponding methods / processes.

[0264] Various embodiments relate to rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is generally considered, given constraints on computational complexity. Rate-distortion optimization is generally formulated as minimizing a rate-distortion function that is a weighted sum of rate and distortion. There are different methods to solve the rate-distortion optimization problem. For example, these methods can be based on extensive testing of all coding options (including all considered modes or coding parameter values) and a complete evaluation of their coding costs and the associated distortion of the reconstructed signal after encoding and decoding. A faster method can also be used to save coding complexity, in particular, to calculate approximate distortion based on a prediction or prediction residual signal rather than on a reconstructed signal. A mixture of these two methods can also be used, such as by using approximate distortion only for some possible coding options and using full distortion for other coding options. Other methods only evaluate a subset of possible coding options. More generally, many methods employ any of a variety of techniques to perform optimization, but the optimization is not necessarily a complete evaluation of both coding costs and associated distortions.

[0265] The embodiments and aspects described herein can be implemented in, for example, methods or processes, devices, software programs, data streams, or signals. Even if only discussed in the context of a single implementation form (e.g., discussed only as a method), the implementation of the features discussed can also be implemented in other forms (e.g., devices or programs). The device can be implemented in, for example, appropriate hardware, software, and firmware. The method can be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, such as, for example, a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0266] Reference to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations thereof means that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations thereof appearing in various places throughout this disclosure are not necessarily all referring to the same embodiment.

[0267] Additionally, the present disclosure may involve “determining” various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0268] Additionally, the present disclosure may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0269] Additionally, the present disclosure may involve "receiving" various information. Like "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, during operations such as, for example, storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is generally involved in one way or another.

[0270] It will be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass selection of only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the case of "A, B, and / or C," and "at least one of A, B, and C," such wording is intended to encompass selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to as many items as listed.

[0271] In addition, as used herein, among other meanings, the word "signal" points to a corresponding decoder to indicate some information. For example, in some embodiments, the encoder signals a specific parameter in a plurality of parameters for region-based filter parameter selection for de-artifact filtering. In this way, in an embodiment, the same parameters are used at both the encoder side and the decoder side. Therefore, for example, the encoder can send (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters and other parameters, signaling (implicit signaling) can be used without transmission to simply allow the decoder to know and select specific parameters. By avoiding the transmission of any actual function, bit saving is achieved in various embodiments. It will be understood that signaling can be done in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to send information to the corresponding decoder with a signal. Although the verb form of the word "signal" is mentioned above, the word "signal" can also be used as a noun in this article.

[0272] Implementations may generate various signals formatted to carry information that may be, for example, stored or transmitted. Information may include, for example, instructions for executing a method or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

[0273] We have described several embodiments. The features of these embodiments may be provided individually or in any combination across the various claim categories and types.

[0274] Note that one or more of the various hardware elements in the described embodiments are referred to as "modules" that implement (i.e., execute, run, etc.) the various functions described herein in conjunction with the corresponding modules. As used herein, a module includes hardware that is considered suitable for a given implementation (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices). Each module described may also include executable instructions for implementing one or more functions described as being implemented by the corresponding module, and it should be noted that those instructions may take the form of or include hardware (i.e., hard-wired) instructions, firmware instructions, software instructions, and / or the like, and may be stored in any suitable non-transitory computer-readable medium or media, such as what is commonly referred to as RAM, ROM, etc.

[0275] Although features and elements are described above in specific combinations, each feature or element may be used alone or in any combination with other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated into a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, buffer memory, semiconductor storage devices, magnetic media of internal hard disks and removable disks, magneto-optical media, and optical media of CD-ROM disks and digital versatile disks. A processor associated with the software may be used to implement a radio frequency transceiver used in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. A method, comprising: obtaining a container file including a plurality of haptic trajectories, the container file including information associating each of the plurality of haptic trajectories with at least one of a corresponding device, a corresponding perception, or a corresponding avatar; obtaining information indicating a selection of at least one device, at least one perception, or at least one avatar; and extracting haptic data in response to the selection, wherein the extracted haptic data excludes at least one haptic trajectory among the plurality of haptic trajectories that is not associated with any of the selected devices, perceptions, or avatars.

2. The method according to claim 1, wherein the method is performed by a server, and the information indicating the selection is received from a client device.

3. The method according to claim 1 or 2, further comprising providing the extracted haptic data to a client device in a bitstream.

4. The method according to any one of claims 1-3, further comprising providing an inventory file indicating at least one available device, at least one available perception, or at least one available avatar, wherein the information indicating the selection is received in response to the inventory file.

5. The method according to any one of claims 1-2, further comprising providing the extracted haptic data to a client device as a container file.

6. The method according to any one of claims 1-5, further comprising rendering the extracted haptic data.

7. The method according to any one of claims 1-6, wherein the information associating a haptic trajectory with a corresponding device includes a device identifier in a haptic channel configuration box associated with the corresponding trajectory.

8. The method according to any one of claims 1-7, wherein the information associating a haptic trajectory with a corresponding perception includes information identifying a group of trajectories, the group of trajectories including a plurality of trajectories associated with the corresponding perception.

9. The method according to any one of claims 1 to 8, wherein the information associating a haptic trajectory with a corresponding avatar includes an avatar identifier in a haptic perception configuration box.

10. The method according to any one of claims 1-9, wherein the container file includes information associating each of the plurality of haptic trajectories with a corresponding device, the selection is a selection of at least one device, and the extracted haptic data excludes at least one haptic trajectory among the plurality of haptic trajectories that is not associated with any of the selected devices.

11. The method according to any one of claims 1-10, wherein the container file is an ISOBMFF file.

12. An apparatus, comprising one or more processors configured to at least perform: obtaining a container file including a plurality of haptic trajectories, the container file including information associating each of the plurality of haptic trajectories with at least one of a corresponding device, a corresponding perception, or a corresponding avatar; obtaining information indicating a selection of at least one device, at least one perception, or at least one avatar; and Haptic data is extracted in response to the selection, wherein the extracted haptic data excludes at least one haptic track of the plurality of haptic tracks that is not associated with any of the selected devices, perceptions, or avatars.

13. The apparatus of claim 12, further configured to provide the extracted haptic data to a client device in a bitstream.

14. The apparatus of claim 12, further comprising providing the extracted haptic data to a client device as a container file.

15. The apparatus of claim 12, further comprising rendering the extracted haptic data.

Images