Haptic frames for handling network congestions and packet loss for streaming of haptic data

Abstract

Apparatus and methods for encoding and decoding haptics experiences. In an example embodiment, information is obtained describing a haptic signal having at least a first band. The haptic signal is encoded in a plurality of haptic streaming units, at least some of the haptic streaming units being associated with respective temporal period, wherein encoding the haptic signal includes encoding the first band of the haptic signal as a plurality of first-band data packets and storing each of the first-band data packets in a respective one of the haptic streaming units. Frame information is encoded that arranges the haptic streaming units in respective frames such that all haptic streaming units in a same frame are associated with the same temporal period.

Description

HAPTIC FRAMES FOR HANDLING NETWORK CONGESTIONS AND PACKET LOSS FOR STREAMING OF HAPTIC DATA CROSS REFERENCE

[0001] This application claims the priority of European Patent Application No.23306543.2, filed 19 September 2023, entitled “Haptic Frames for Handling Network Congestions and Packet Loss for Streaming of Haptic Data,” which is incorporated herein by reference in its entirety. BACKGROUND

[0002] The present disclosure relates to encoding, delivery, decoding, and processing of haptics data.

[0003] Haptics refers to the sense of touch and includes two dimensions, tactile and kinesthetic. The first one relates to tactile sensations such as friction, roughness, hardness, temperature and is felt through the mechanoreceptors of the skin (Merkel cell, Ruffini ending, Meissner corpuscle, Pacinian corpuscle), and thermoreceptors. The second one is linked to the sensation of force / torque, position, motion / velocity provided by the muscles, tendons, and the mechanoreceptors in the joints.

[0004] Haptics can be considered as a top-level medium such as images or audio. It is now part of the MPEG standardization process that standardizes a haptic codec and associated information. With the on-going MPEG standardization process on the coded representation of Haptics, a new Haptic coding format is being defined. SUMMARY

[0005] A haptic decoding method according to some embodiments comprises: obtaining an encoded haptic signal including a plurality of haptic streaming units; obtaining frame information arranging the haptic streaming units in respective frames, the frame information including a respective timing packet in each of a plurality of the haptic streaming units; for at least a plurality of the haptic streaming units, determining a temporal period associated with the respective haptic streaming unit based on the timing packets; and decoding at least a first band of the encoded haptic signal from a plurality of first-band data packets, each of the first-band data packets being obtained from a respective one of the haptic streaming units.

[0006] A haptic decoding apparatus according to some embodiments comprises one or more processors configured to perform at least: obtaining an encoded haptic signal including a pluralityof haptic streaming units; obtaining frame information arranging the haptic streaming units in respective frames, the frame information including a respective timing packet in each of a plurality of the haptic streaming units; for at least a plurality of the haptic streaming units, determining a temporal period associated with the respective haptic streaming unit based on the timing packets; and decoding at least a first band of the encoded haptic signal from a plurality of first-band data packets, each of the first-band data packets being obtained from a respective one of the haptic streaming units.

[0007] A haptic encoding method according to some embodiments comprises: obtaining information describing a haptic signal having at least a first band; encoding the haptic signal in a plurality of haptic streaming units, at least some of the haptic streaming units being associated with a respective temporal period, wherein encoding the haptic signal includes encoding the first band of the haptic signal as a plurality of first-band data packets and storing each of the first-band data packets in a respective one of the haptic streaming units; and encoding frame information arranging the haptic streaming units in respective frames such that all haptic streaming units in a same frame are associated with the same temporal period, the frame information including a respective timing packet in each of a plurality of the haptic streaming units.

[0008] A haptic encoding apparatus according to some embodiments comprises one or more processors configured to perform at least: obtaining information describing a haptic signal having at least a first band; encoding the haptic signal in a plurality of haptic streaming units, at least some of the haptic streaming units being associated with a respective temporal period, wherein encoding the haptic signal includes encoding the first band of the haptic signal as a plurality of first-band data packets and storing each of the first-band data packets in a respective one of the haptic streaming units; and encoding frame information arranging the haptic streaming units in respective frames such that all haptic streaming units in a same frame are associated with the same temporal period, the frame information including a respective timing packet in each of a plurality of the haptic streaming units.

[0009] A method according to some embodiments comprises: obtaining information describing a haptic signal having at least a first band; encoding the haptic signal in a plurality of haptic streaming units, at least some of the haptic streaming units being associated with respective temporal period, wherein encoding the haptic signal includes encoding the first band of the haptic signal as a plurality of first-band data packets and storing each of the first-band data packets in a respective one of the haptic streaming units; and encoding frame information arranging the haptic streaming units in respective frames such that all haptic streaming units in a same frame are associated with the same temporal period.

[0010] A method according to some embodiments comprises: obtaining an encoded haptic signal including a plurality of haptic streaming units; obtaining frame information arranging the haptic streaming units in respective frames; for at least a plurality of the haptic streaming units, determining a temporal period associated with the respective haptic streaming unit based at leastin part on the frame information; and decoding at least a first band of the encoded haptic signal from a plurality of first-band data packets, each of the first-band data packets being obtained from a respective one of the haptic streaming units.

[0011] In some embodiments, the frame information includes one or more of the following: time stamp information provided in a header of respective haptic streaming units; time stamp information provided in a timing packet in respective haptic streaming units; sequence information provided in a header of respective haptic streaming units; sequence information provided in a timing packet in respective haptic streaming units; at least one start packet indicating a start of a frame; and / or at least one end packet indicating an end of a frame.

[0012] Encoder and decoder apparatus are provided to perform the methods described herein. An encoder or decoder apparatus may include a processor configured to perform the methods described herein. The apparatus may include a computer-readable medium (e.g. a non-transitory medium) storing instructions for performing the methods described herein. In some embodiments, a computer-readable medium (e.g. a non-transitory medium) stores haptic data encoded using any of the methods described herein.

[0013] One or more of the present embodiments also provide a computer readable storage medium having stored thereon instructions for performing bi-directional optical flow, encoding or decoding data according to any of the methods described above. The present embodiments also provide a computer readable storage medium having stored thereon a bitstream generated according to the methods described above. The present embodiments also provide a method and apparatus for transmitting the bitstream generated according to the methods described above. The present embodiments also provide a computer program product including instructions for performing any of the methods described. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The following detailed description will be better understood when read in conjunction with the appended drawings, in which there are shown examples of one or more of the multiple embodiments of the present disclosure. It should be understood, however, that the embodiments described herein are not limited to the precise arrangements and instrumentalities shown in the drawings.

[0015] FIG.1A is a system diagram illustrating an example communications system in which one or more disclosed embodiments may be implemented.

[0016] FIG. 1B is a system diagram illustrating an example wireless transmit / receive unit (WTRU) that may be used within the communications system illustrated in FIG.1A according to an embodiment.

[0017] FIG.1C is a functional block diagram of a system used in some embodiments described herein.

[0018] FIG. 1D illustrates an MPEG haptics data structure that may be used in some embodiments.

[0019] FIG. 2 illustrates an example of a haptic signal (top) and its decomposition in two frequency bands (bottom).

[0020] FIG.3 illustrates an example of haptic data stored on two channels. The first channel contains a curve band and a vectorial band. The second channel contains a single curve band.

[0021] FIG.4 illustrates an example MPEG Haptic codec architecture.

[0022] FIG.5 illustrates an example MIHS packetization.

[0023] FIG.6 illustrates another example of MIHS packetization.

[0024] FIG.7 illustrates an example of streaming of MIHS data.

[0025] FIG.8 illustrates an example of packet loss in the streaming process.

[0026] FIG.9 illustrates an example of results of packet loss on an MIHS stream.

[0027] FIG.10 illustrates a simplified example of an MIHS Unit Stream.

[0028] FIG.11 illustrates an example of packetization using the concept of MIHS frames.

[0029] FIG.12 illustrates MIHS packetization as performed in Example 1.

[0030] FIG.13 illustrates adding timing information to MIHS units without constraints.

[0031] FIG.14 illustrates MIHS packetization as performed in Example 2.

[0032] FIG.15 illustrates MIHS packetization as performed in Example 3.

[0033] FIG.16 illustrates MIHS packetization as performed in Example 4.

[0034] FIG.17 illustrates MIHS unit packetization using Start and End units.

[0035] FIG.18 illustrates a packet loss example based on Example 4.

[0036] FIG. 19 illustrates packetization of MIHS units based on Example 4, with information indicating the duration of frames provided in the Initialization MIHS Unit.

[0037] FIG. 20 illustrates packetization of MIHS units based on Embodiment 4, with priority information.

[0038] FIG.21 illustrates an MIHS unit structure. EXAMPLE NETWORKS FOR IMPLEMENTATION OF THE EMBODIMENTS

[0039] FIG.1A is a diagram illustrating an example communications system 100 in which one or more disclosed embodiments may be implemented. The communications system 100 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communications system 100 may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communications systems 100 may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access(TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block-filtered OFDM, filter bank multicarrier (FBMC), and the like.

[0040] As shown in FIG. 1A, the communications system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, a RAN 104, a CN 106, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, though it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d, any of which may be referred to as a “station” and / or a “STA”, may be configured to transmit and / or receive wireless signals and may include a user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smartphone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (IoT) device, a watch or other wearable, a head-mounted display (HMD), a vehicle, a drone, a medical device and applications (e.g., remote surgery), an industrial device and applications (e.g., a robot and / or other wireless devices operating in an industrial and / or an automated processing chain contexts), a consumer electronics device, a device operating on commercial and / or industrial wireless networks, and the like. Any of the WTRUs 102a, 102b, 102c and 102d may be interchangeably referred to as a UE.

[0041] The communications systems 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN 106, the Internet 110, and / or the other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a gNB, a NR NodeB, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each depicted as a single element, it will be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0042] The base station 114a may be part of the RAN 104, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and / or the base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage for a wireless service to a specific geographical area that may be relatively fixed or that may change over time. The cell may further be divided into cell sectors. For example, the cellassociated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e., one for each sector of the cell. In an embodiment, the base station 114a may employ multiple-input multiple output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in desired spatial directions.

[0043] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0044] More specifically, as noted above, the communications system 100 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, the base station 114a in the RAN 104 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 116 using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High-Speed Downlink (DL) Packet Access (HSDPA) and / or High-Speed UL Packet Access (HSUPA).

[0045] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE- A) and / or LTE-Advanced Pro (LTE-A Pro).

[0046] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as NR Radio Access, which may establish the air interface 116 using New Radio (NR).

[0047] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may implement LTE radio access and NR radio access together, for instance using dual connectivity (DC) principles. Thus, the air interface utilized by WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., a eNB and a gNB).

[0048] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA20001X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), InterimStandard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and the like.

[0049] The base station 114b in FIG.1A may be a wireless router, Home Node B, Home eNode B, or access point, for example, and may utilize any suitable RAT for facilitating wireless connectivity in a localized area, such as a place of business, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a roadway, and the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In an embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular- based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR etc.) to establish a picocell or femtocell. As shown in FIG.1A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not be required to access the Internet 110 via the CN 106.

[0050] The RAN 104 may be in communication with the CN 106, which may be any type of network configured to provide voice, data, applications, and / or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have varying quality of service (QoS) requirements, such as differing throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, and the like. The CN 106 may provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, etc., and / or perform high-level security functions, such as user authentication. Although not shown in FIG.1A, it will be appreciated that the RAN 104 and / or the CN 106 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 104 or a different RAT. For example, in addition to being connected to the RAN 104, which may be utilizing a NR radio technology, the CN 106 may also be in communication with another RAN (not shown) employing a GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0051] The CN 106 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or the other networks 112. The PSTN 108 may include circuit-switched telephone networks that provide plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the transmission control protocol (TCP), user datagram protocol (UDP) and / or the internet protocol (IP) in the TCP / IP internet protocol suite. The networks 112 may include wired and / or wireless communications networks owned and / or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs, which may employ the same RAT as the RAN 104 or a different RAT.

[0052] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links). For example, the WTRU 102c shown in FIG.1A may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[0053] FIG.1B is a system diagram illustrating an example WTRU 102. As shown in FIG.1B, the WTRU 102 may include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138, among others. It will be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0054] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. While FIG.1B depicts the processor 118 and the transceiver 120 as separate components, it will be appreciated that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0055] The transmit / receive element 122 may be configured to transmit signals to, or receive signals from, a base station (e.g., the base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In an embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF and light signals. It will be appreciated that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0056] Although the transmit / receive element 122 is depicted in FIG.1B as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0057] The transceiver 120 may be configured to modulate the signals that are to be transmitted by the transmit / receive element 122 and to demodulate the signals that are received by the transmit / receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11, for example.

[0058] The processor 118 of the WTRU 102 may be coupled to, and may receive user input data from, the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 130 and / or the removable memory 132. The non-removable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from, and store data in, memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[0059] The processor 118 may receive power from the power source 134, and may be configured to distribute and / or control the power to the other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel- zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0060] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or in lieu of, the information from the GPS chipset 136, the WTRU 102 may receive location information over the air interface 116 from a base station (e.g., base stations 114a, 114b) and / or determine its location based on the timing of the signals being received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment.

[0061] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photographs and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a Virtual Reality and / orAugmented Reality (VR / AR) device, an activity tracker, and the like. The peripherals 138 may include one or more sensors, the sensors may be one or more of a gyroscope, an accelerometer, a hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0062] The WTRU 102 may include a full duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for both the UL (e.g., for transmission) and downlink (e.g., for reception) may be concurrent and / or simultaneous. The full duplex radio may include an interference management unit to reduce and or substantially eliminate self-interference via either hardware (e.g., a choke) or signal processing via a processor (e.g., a separate processor (not shown) or via processor 118). In an embodiment, the WRTU 102 may include a half-duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for either the UL (e.g., for transmission) or the downlink (e.g., for reception)).

[0063] Although the WTRU is described in FIGs. 1A-1B as a wireless terminal, it is contemplated that in certain representative embodiments that such a terminal may use (e.g., temporarily or permanently) wired communication interfaces with the communication network.

[0064] In representative embodiments, the other network 112 may be a WLAN.

[0065] In view of FIGs.1A-1B, and the corresponding description , one or more, or all, of the functions described herein may be performed by one or more emulation devices (not shown). The emulation devices may be one or more devices configured to emulate one or more, or all, of the functions described herein. For example, the emulation devices may be used to test other devices and / or to simulate network and / or WTRU functions.

[0066] The emulation devices may be designed to implement one or more tests of other devices in a lab environment and / or in an operator network environment. For example, the one or more emulation devices may perform the one or more, or all, functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network in order to test other devices within the communication network. The one or more emulation devices may perform the one or more, or all, functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. The emulation device may be directly coupled to another device for purposes of testing and / or may performing testing using over-the-air wireless communications.

[0067] The one or more emulation devices may perform the one or more, including all, functions while not being implemented / deployed as part of a wired and / or wireless communication network. For example, the emulation devices may be utilized in a testing scenario in a testing laboratory and / or a non-deployed (e.g., testing) wired and / or wireless communication network in order to implement testing of one or more components. The one or more emulation devices may be testequipment. Direct RF coupling and / or wireless communications via RF circuitry (e.g., which may include one or more antennas) may be used by the emulation devices to transmit and / or receive data. Example Systems.

[0068] The embodiments described herein are not limited to being implemented on a WTRU. Such embodiments may be implemented using other systems, such as the system of FIG.1C. FIG.1C is a block diagram of an example of a system in which various aspects and embodiments are implemented. System 1000 can be embodied as a device including the various components described below and is configured to perform one or more of the aspects described in this document. Examples of such devices, include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 1000, singly or in combination, can be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, the system 1000 is communicatively coupled to one or more other systems, or other electronic devices, via, for example, a communications bus or through dedicated input and / or output ports. In various embodiments, the system 1000 is configured to implement one or more of the aspects described in this document.

[0069] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing, for example, the various aspects described in this document. Processor 1010 can include embedded memory, input output interface, and various other circuitries as known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device, and / or a non-volatile memory device). System 1000 includes a storage device 1040, which can include non-volatile memory and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drive, and / or optical disk drive. The storage device 1040 can include an internal storage device, an attached storage device (including detachable and non-detachable storage devices), and / or a network accessible storage device, as non-limiting examples.

[0070] System 1000 includes an encoder / decoder module 1030 configured, for example, to process data to provide an encoded haptic or decoded haptic, and the encoder / decoder module 1030 can include its own processor and memory. The encoder / decoder module 1030 represents module(s) that can be included in a device to perform the encoding and / or decoding functions. As is known, a device can include one or both of the encoding and decoding modules.Additionally, encoder / decoder module 1030 can be implemented as a separate element of system 1000 or can be incorporated within processor 1010 as a combination of hardware and software as known to those skilled in the art.

[0071] Program code to be loaded onto processor 1010 or encoder / decoder 1030 to perform the various aspects described in this document can be stored in storage device 1040 and subsequently loaded onto memory 1020 for execution by processor 1010. In accordance with various embodiments, one or more of processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 can store one or more of various items during the performance of the processes described in this document. Such stored items can include, but are not limited to, the input haptic, the decoded haptic or portions of the decoded haptic, the bitstream, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0072] In some embodiments, memory inside of the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and to provide working memory for processing that is needed during encoding or decoding. In other embodiments, however, a memory external to the processing device (for example, the processing device can be either the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory can be the memory 1020 and / or the storage device 1040, for example, a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, an external non-volatile flash memory is used to store the operating system of, for example, a television. In at least one embodiment, a fast external dynamic volatile memory such as a RAM is used as working memory for video coding and decoding operations, such as for MPEG-2 (MPEG refers to the Moving Picture Experts Group, MPEG-2 is also referred to as ISO / IEC 13818, and 13818- 1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by JVET, the Joint Video Experts Team).

[0073] The input to the elements of system 1000 can be provided through various input devices as indicated in block 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Component (COMP) input terminal (or a set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Other examples, not shown in FIG.1C, include composite video.

[0074] In various embodiments, the input devices of block 1130 have associated respective input processing elements as known in the art. For example, the RF portion can be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) downconverting the selected signal, (iii) band-limiting again to a narrower band of frequencies to select (for example) a signal frequency band which can be referred to as a channel in certain embodiments, (iv) demodulating thedownconverted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets. The RF portion of various embodiments includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion can include a tuner that performs various of these functions, including, for example, downconverting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing element receives an RF signal transmitted over a wired (for example, cable) medium, and performs frequency selection by filtering, downconverting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements performing similar or different functions. Adding elements can include inserting elements in between existing elements, such as, for example, inserting amplifiers and an analog-to-digital converter. In various embodiments, the RF portion includes an antenna.

[0075] Additionally, the USB and / or HDMI terminals can include respective interface processors for connecting system 1000 to other electronic devices across USB and / or HDMI connections. It is to be understood that various aspects of input processing, for example, Reed-Solomon error correction, can be implemented, for example, within a separate input processing IC or within processor 1010 as necessary. Similarly, aspects of USB or HDMI interface processing can be implemented within separate interface ICs or within processor 1010 as necessary. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 1010, and encoder / decoder 1030 operating in combination with the memory and storage elements to process the datastream as necessary for presentation on an output device.

[0076] Various elements of system 1000 can be provided within an integrated housing, Within the integrated housing, the various elements can be interconnected and transmit data therebetween using suitable connection arrangement 1140, for example, an internal bus as known in the art, including the Inter-IC (I2C) bus, wiring, and printed circuit boards.

[0077] The system 1000 includes communication interface 1050 that enables communication with other devices via communication channel 1060. The communication interface 1050 can include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 1060. The communication interface 1050 can include, but is not limited to, a modem or network card and the communication channel 1060 can be implemented, for example, within a wired and / or a wireless medium.

[0078] Data is streamed, or otherwise provided, to the system 1000, in various embodiments, using a wireless network such as a Wi-Fi network, for example IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal of these embodiments isreceived over the communications channel 1060 and the communications interface 1050 which are adapted for Wi-Fi communications. The communications channel 1060 of these embodiments is typically connected to an access point or router that provides access to external networks including the Internet for allowing streaming applications and other over-the-top communications. Other embodiments provide streamed data to the system 1000 using a set-top box that delivers the data over the HDMI connection of the input block 1130. Still other embodiments provide streamed data to the system 1000 using the RF connection of the input block 1130. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, for example a cellular network or a Bluetooth network.

[0079] The system 1000 can provide an output signal to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. The display 1100 of various embodiments includes one or more of, for example, a touchscreen display, an organic light- emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1100 can be for a television, a tablet, a laptop, a cell phone (mobile phone), or other device. The display 1100 can also be integrated with other components (for example, as in a smart phone), or separate (for example, an external monitor for a laptop). The other peripheral devices 1120 include, in various examples of embodiments, one or more of a stand-alone digital video disc (or digital versatile disc) (DVR, for both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide a function based on the output of the system 1000. For example, a disk player performs the function of playing the output of the system 1000.

[0080] In various embodiments, control signals are communicated between the system 1000 and the display 1100, speakers 1110, or other peripheral devices 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communications protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, the output devices can be connected to system 1000 using the communications channel 1060 via the communications interface 1050. The display 1100 and speakers 1110 can be integrated in a single unit with the other components of system 1000 in an electronic device such as, for example, a television. In various embodiments, the display interface 1070 includes a display driver, such as, for example, a timing controller (T Con) chip.

[0081] The display 1100 and speaker 1110 can alternatively be separate from one or more of the other components, for example, if the RF portion of input 1130 is part of a separate set-top box. In various embodiments in which the display 1100 and speakers 1110 are external components, the output signal can be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0082] The embodiments can be carried out by computer software implemented by the processor 1010 or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. The memory 1020 can be of any type appropriate to the technical environment and can be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non- limiting examples. The processor 1010 can be of any type appropriate to the technical environment, and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as non-limiting examples. DETAILED DESCRIPTION Overview of haptic data coding. MPEG Haptic Data Structure.

[0083] Example embodiments make use of a data structure, such as an MPEG haptic data structure, that follows a hierarchical organization as illustrated in FIG.1D.

[0084] The format first contains some high-level metadata information regarding the overall haptic experience defined in the file. It then provides a list of avatars (e.g., body representations) later referenced in the file to specify the desired location of haptic stimuli on the body. The haptic data itself is then described through a list of perceptions. These perceptions correspond to haptic signals associated with specific perception modalities, such as vibration, force, position, velocity, temperature, among other possibilities.

[0085] A perception contains a list of tracks where the data is decomposed in frequency bands.

[0086] Each band provides information describing part of the signal in a given frequency range. The band itself is described with a list of haptic effects, each containing a list of keyframes. The haptic signal in a track can then be reconstructed by combining the data in the different bands. FIG.2 illustrates an example of how a haptic signal can be decomposed in two frequency bands. By adding the high and low frequency bands, the original signal can be reconstructed.

[0087] An example format used in some embodiments accommodates four types of haptic bands: transient bands, curve bands, vectorial bands and wavelet bands. Each band is composed of a series of effects, and each effect is described by a list of keyframes. The data contained in the effects and keyframes is interpreted differently for different types of haptic bands and encoding modalities.

[0088] For a transient band, each effect stores a set of keyframes describing a position, an amplitude, and a frequency. A keyframe represents a transient event.

[0089] For a curve band, each effect stores a set of keyframes defining a position and an amplitude. The keyframes represent the control points of the curve. The type of interpolation function (e.g. cubic or linear) used to generate the band is specified in the metadata of the band.

[0090] For vectorial bands, the effect stores a set of keyframes describing a position, an amplitude and a frequency.

[0091] For wavelet bands, the effect stores the contents of one wavelet block. It contains a keyframe for every coefficient of the wavelet transformed and quantized signal, with only the amplitude value used. The coefficients may be scaled to a range of [-1,1]. Additionally, the original maximum amplitude may be stored in a keyframe, as well as the maximum number of used bits.

[0092] FIG. 3 illustrates an example of haptic data containing a single haptic perception with two channels. The first channel contains a curve band with a single effect and a vectorial band with four effects. The second channel contains a single curve band. MPEG Codec.

[0093] The overall architecture of an example codec, such as an MPEG codec, that may be used in implementing some embodiments is illustrated in FIG. 4. The encoded signal can be exported in a format such as a JSON (human readable) interchange format or as a compressed binary distribution format.

[0094] The encoding process can take as its input different formats such as descriptive haptics files (.IVS and .AHAP) or waveform PCM files (.wav). The encoding of these two types of file formats follows two distinct approaches. For descriptive input files, the encoder first analyzes the data and then transcodes it directly to the MPEG format. For waveform signals, the data is processed using more common signal analysis methods.

[0095] The present disclosure addresses example embodiments relating to encoding and decoding of the waveform PCM signals. This process is described in more detail in FIG.4.

[0096] Given an input PCM signal, an example encoding process starts by performing a frequency band decomposition. Using a low pass filter and a high pass filter, it splits the signal into a low frequency band and a high frequency band. The two bands are then processed differently. Various methods can be used for the encoding of the high frequency. One solution is to split the high frequency signal into smaller fixed length windows and use Short-time Fourier Transform (STFT) to decompose the signal in the frequency spectrum. Another solution (the one currently used in the MPEG reference encoder) consists in using wavelet transforms to encode the high frequencies.

[0097] As detailed in the previous section and illustrated in FIG.1D, the data structure allows for the definition of multiple bands with different frequency ranges. These bands may be used to store the coefficients of the Fourier or wavelet transforms.MIHS Format.

[0098] The MPEG Haptic standard defines two different file formats to encode this data structure: an interchange file format (.hjif) encoded in JSON, and a binary encoded streaming format (MIHS) that can also be stored as a binary file (.hmpg). The two formats have complementary purposes, and a lossy one-to-one conversion can be operated between them.

[0099] For the sake of explanation, the present disclosure makes reference to the use of the MPEG-I Haptic Stream (MIHS) format. This format was designed for distribution and streaming purposes. It packetizes the data to optimize transport. However, the principles described herein are not limited to the use of the MIHS format.

[0100] The packets are of variable duration and include two levels of packetization. At a higher level, an MIHS unit covers a duration of time and includes zero or more MIHS packets. At a lower level, an MIHS packet includes metadata or haptic effect data. FIG.5 schematically illustrates the high-level structure of MIHS packetization.

[0101] Conventionally, each MIHS unit covers a nonoverlapping duration of haptic presentation time. For example, the unit starts at the end of the previous MIHS unit and covers the duration of time defined by its duration field. The MIHS unit is followed by the next MIHS unit, unless it is the last MIHS unit of the haptic experience. All MIHS packets of an MIHS unit include information indicating the starting time and duration of the containing MIHS unit.

[0102] An MIHS unit may have one of the following types: initialization, temporal, spatial, or slient.

[0103] Initialization. An initialization unit contains one timing MIHS packet and may include one or more metadata MIHS packets. The initialization unit starts a new time anchor in haptics presentation time by setting the timestamp and may also set or change the timescale. The duration of an initialization unit is zero.

[0104] Temporal. A temporal unit contains one or more MIHS packets. The duration of a temporal unit is a positive number.

[0105] Spatial. A spatial unit contains one or more MIHS packets. The duration of a spatial unit is zero. If compatibility with the HJIF format is desired, all effects of a spatial modality are included in the first MIHS data unit.

[0106] Silent. A silent unit indicates that there is no effect that starts during this time interval, and it does not include any MIHS packets. The duration of a silent unit is constrained to be a positive number.

[0107] Constraints may be applied such that the first MIHS unit in a haptic stream is an initialization unit.

[0108] An MIHS unit may be a sync unit or a non-sync unit. A sync unit resets the previous effects and therefore provides an independent haptic experience from the previous MIHS units.A non-sync unit is the continuation of previous MIHS units and cannot be independently decoded and rendered without decoding the previous MIHS unit(s).

[0109] FIG. 6 schematically illustrates an example of how the effect of FIG. 3 could be packetized. In this example, for each MIHS unit, MIHS packets would be created for each band of each channel for the duration of the MIHS unit. An MIHS unit contains all the haptic data defined in a given time range. In addition, an MIHS initialization unit would be created to store timing and metadata packets. In this example, an MIHS packet from a given unit may depend on MIHS packets from the previous unit. For instance, MIHS packet 4 may depend on MIHS packet 1 and MIHS packet 7 may depend on MIHS packets 4 and 1.

[0110] FIG.7 illustrates an example of how the data defined in MIHS units may be streamed. For the sake of simplicity, the example shows one possible solution where a single MIHS unit is used to build a streaming packet. Other solutions may aggregate MIHS unit together if the payload is low or may fragment them if the payload is too important. The identified issues and the proposed solutions presented in this disclosure apply to all of these methods. FIG.7 illustrates that MIHS units are transported over IP packets, using media transport protocols such as Real-Time Streaming Protocol (RTSP), Real-Time Protocol (RTP), or MPEG-Dash for instance. Various different packets may be used in different implementations, and each such packet may be referred to herein as a media transport packet, IP packet, RTP packet, media packet, or streaming packet to designate packets used to transmit MIHS units over a network. Issues Addressed in Some Embodiments.

[0111] Conventionally, MIHS units are constrained so as not to overlap in time. This constraint implies that each MIHS unit contains all the haptic data for the time range it covers. In addition, only MIHS initialization units contain timing information (through timing packets). Silent and temporal MIHS units only define a duration. The starting time of a unit is then defined by the starting time of the previous initialization Unit and the duration of all MIHS units in between. Such a constraint imposed in MIHS presents at least two problems. First, it does not allow for proper handling of the loss of an MIHS unit during transmission (e.g., due to an RTP packet loss). Second, it is not adapted to handle congestion issues through differentiated handling. Packet Loss.

[0112] One issue with the conventional MIHS structure is that it is not optimal to handle MIHS unit loss that may occur during streaming. A reason for this is that some media transport protocols do not guarantee media packet delivery. For example, RTP typically does not retransmit lost media packets, and while RTP supports retransmitting lost media packets, this is not a mandatory feature of the RTP protocol. FIG. 8 schematically illustrates an example of packet loss in the streaming process, resulting in the loss of an MIHS unit. In this example, the third streaming packet containing the MIHS Unit 2 is lost during the streaming process. Packet loss may happen for a variety of reasons including poor network quality, network congestion, hardware issues orsoftware issues. Some example embodiments described herein provide for adaptations to the MIHS format to ease the handling this type of issues.

[0113] FIG.9 illustrates the impact of losing a packet during the streaming of the data. A first problem caused by this packet loss is that, since a single conventional MIHS unit contains all the haptic data in a given time range, losing an MIHS unit results in the loss of all haptic data for this time range. This can impact the quality of experience for the end user, since this data may be replaced by either a haptic silence or by an estimated rendering, which can be erroneous and damaging to the experience.

[0114] Another problem appears in conventional systems, in which a temporal MIHS unit does not contain timing information but only defines the duration of the unit. The starting time of a unit in such systems is based on the timestamp specified in the previous MIHS initialization unit and the duration of the MIHS units in between. Losing an MIHS unit therefore implies losing the timing information of the subsequent MIHS units. As illustrated in FIG. 9, it would be impossible to properly assess the starting time of the MIHS Unit 3. The lack of timing information at the MIHS level may also be problematic for the detection of the MIHS unit loss itself. By default, if the receiver is not aware of an MIHS unit loss, it may incorrectly infer that MIHS Unit 3 is simply following MIHS Unit 1. This can diminish the quality of experience for the end user, since the haptic rendition may become out of sync with other, e.g., audio / video inputs.

[0115] Overall, the lack of timing information in most MIHS units and the non-overlapping constraint of MIHS units are problematic to properly address packet loss with MIHS streams and may result in synchronization issues with other media, for instance. Furthermore, the haptics decoder typically does not have access to lower layer timing information (e.g., the RTP timestamp), so if an MIHS unit were lost in the streaming process, the decoder would not be able to detect missing data and infer the duration of missing MIHS units (and therefore the starting time of the following unit). Network Congestion.

[0116] Another issue of the conventional structure of the MIHS data relates to network congestion handling. When streaming data, it is quite common to encounter congestion issues on the network. To address the issue, the streamer may choose to reduce the quality of the streamed data by reducing the number of packets sent through the network. As illustrated in FIG. 9, in the case of MIHS units, not sending a packet implies the loss of all haptic data within the corresponding time range. In addition to the impact on timing of the haptic rendition, as described above, there is therefore additionally a lack of mechanism enabling a graceful degradation of the rendition. It would be desirable to provide a mechanism to enable prioritizing haptics data within a given time range so as to allow for a degraded rendering. The current MPEG Haptic standard does not have such a mechanism. Example embodiments described herein provide a mechanism to handle congestion gracefully by limiting the amount of data sent for a given time range.Overview of Example Embodiments.

[0117] To address one or more of the issues described above, various embodiments are described herein. While the embodiments are described here for the sake of clarity as modifications to the current MPEG Haptic Specification, the principles described herein are not limited to use in that specification. Some example embodiments provide modifications that affect the behavior of endpoints sending or receiving haptics streams. Some example embodiments change part of the bitstream used to convey haptics data. Some example embodiments further remove constraints as compared to the current MPEG Haptic Specification.

[0118] For the sake of clarity, some embodiments described herein are illustrated with the example given in FIG.6. FIG.10 is a simplified representation of this example and will be used as reference for further descriptions.

[0119] In some example embodiments, in order to handle congestion issues, haptics data is provided using a format that does not require packets to be nonoverlapping. This feature may be applied to any or all of the embodiments described herein. In embodiments based on MIHS units, this may be implemented by removing the existing nonoverlapping constraint. In addition to removing the nonoverlapping constraint of MIHS units, any or all embodiments described herein may include timing information (which may be provided in various formats) in each MIHS unit to address packet loss and congestion issues. Different embodiments may use different techniques for providing timing information in addition to the duration information. In terms of the existing MIHS framework, example embodiments introduce the concept of frames (which may be referred to as MIHS frames or MIHS samples, the two can be used interchangeably) that cover consecutive and nonoverlapping time ranges, with each frame containing one or more MIHS units that cover the same time span. In example embodiments, MIHS frames may contain multiple temporal and / or silent MIHS units. The concept is illustrated in FIG.11.

[0120] The use of frames (such as MIHS frames) as described herein may improve processing in case of congestion issues. With this approach, MIHS packets that were in the same MIHS unit in the current format (see FIG.10) can now be placed in one of several MIHS units that are in the same MIHS frame, as shown in FIGs. 12-15. In case of congestion, this allows the sender to reduce the amount of data sent through the network by sending less data (e.g. by sending fewer MIHS units) for a given time range. With this approach, the MIHS packets in a same time range may be packetized into different MIHS units (covering the same time range). The specific method used to group packets (e.g., called “packetization” herein) in different MIHS units of the same MIHS frame are application dependent and can be implemented at the authoring and / or encoding stage. In some embodiments, packetization may be based on band frequency ranges for instance, where all haptic data in a given MIHS unit share the same frequency ranges. This may be useful for instance to reduce the amount of data sent through the network by selecting only MIHS units containing high frequency data or low frequency data. Other embodiments may packetize all MIHS packets containing data from the same channel in a single MIHS unit. For thesake of explanation, this method of packetizing based on channel is used to illustrate different embodiments below (see FIGs.12-17), but it should be understood that these embodiments are not limited to the use of a particular packetization technique.

[0121] In some embodiments, a sender may place all MIHS units corresponding to certain band frequency ranges, which are known to the sender to contribute the most to the quality of experience for end users, depending on the application type and / or on player devices.

[0122] In some embodiments, a sender may determine that certain channels contribute the most to the quality of experience for end users as they contain the haptic data to be rendered on the haptic actuators / devices contributing the most to the haptic experience. For instance, in a given haptic experience using multiple devices (such as a haptic vest, a haptic enabled headset, vibrating game controllers, …), the data for each device may be grouped in dedicated MIHS units. In case of network congestion, this allows the sender to reduce the size of the stream by sending only the MIHS units of the main haptic device (the haptic vest for instance) in a given time range.

[0123] In some embodiments (which may be implemented in conjunction with the embodiments described above), the encoder may set the MIHS layer of all MIHS packets grouped in a first MIHS unit to layer 0. The encoder may set the MIHS layer of all MIHS packets grouped in a second unit to layer 1, and so on. The sender may use the MIHS layer to determine whether to send an MIHS unit, or not. For example, when experiencing congestion (e.g., a long sending queue), the sender may decide to send layer 0 MIHS unit in priority.

[0124] In some embodiments (which may be implemented in conjunction with the embodiments described above), priority of the MIHS units may be set in the MIHS unit itself by adding the information in the MIHS unit header as detailed above regarding the addition of layer and priority information.

[0125] For the sake of clarity, different categories of embodiments are described below in different sections, but it should be understood that features of these different embodiments can be combined into additional embodiments. Embodiments using additional timing information.

[0126] Some embodiments restructure the MIHS data to add additional timing information to the MIHS unit. This timing information may be absolute, or it may be relative to the timing information of the previous initialization unit. Two specific examples of such embodiments are described herein, though other examples also fall within the scope of the present disclosure. With this additional timing information added and the removal of the nonoverlapping constraint of MIHS units, example embodiments allow for more than one MIHS unit to be defined in the same time range (e.g. in the same haptic frame). In the example of FIG.12, for instance, all the data packets from the same channel are gathered in a single MIHS unit; MIHS Units 1, 3 and 5 contain the data from the channel 1 and MIHS Units 2, 4 and 6 contain the data from the channel 2. MIHSUnits 1 and 2 belong to the same MIHS Frame (e.g. cover the same time range), MIHS Units 3 and 4 belong to the same MIHS Frame and MIHS Units 5 and 6 belong to the same MIHS frame.

[0127] This first embodiment not only allows handling of congestion issues, but it also allows for handling of packet loss issues. Indeed, even if previous MIHS units are lost, the receiver will be able to properly synchronize the new packets. From the decoder side, it will also allow the identification of missing data and eventually cope with it through various methods (interpolation for instance). Example 1.

[0128] Example 1 is an implementation of an embodiment using additional timing information. In this example, timing information is added through additional bits in the MIHS header, as illustrated with the added timestamp property shown in FIG.12.

[0129] In this example, additional constraints may also be imposed. For example, only adding timing information for each unit and removing the nonoverlapping constraint may result in unstructured timed data that would not be optimal for network packet creation and steaming in general (see FIG.13). To cope with this issue, the haptics encoder and decoder are configured in some embodiments to include additional constraints on their behavior. The following additional constraints may be used to ensure that all MIHS units are properly aligned and organized in MIHS Frames where all the MIHS units in the same MIHS frame share the same timestamp and duration (as illustrated in FIG.13). ^ If the last non-spatial MIHS unit before a temporal or a silent MIHS unit is an initialization unit, the timestamp of the temporal or silent MIHS unit is constrained to be equal to the timestamp of the timing packet in the MIHS initialization unit ^ If the last non-spatial MIHS unit before a second temporal or a silent MIHS unit is a first temporal or a silent MIHS unit, the timestamp of the second MIHS unit is constrained to be equal to the timestamp of the first temporal or a silent MIHS unit or equal to the timestamp of the first temporal or a silent MIHS unit plus the duration of the first temporal or a silent MIHS unit. ^ The duration of all MIHS units with the same timestamp is constrained to be equal.

[0130] For spatial units and initialization units, this added information in the MIHS unit header can be ignored.

[0131] An embodiment such as that of Example 1 may be implemented using a modification of the MPEG Haptic standard. For example, modifications may be performed on some of the constraints set forth in the specification of the standard. Components of a decoder that read the header of MIHS units may also be modified. In an example implementation, the MPEG Haptic standard is modified with the addition of new material as shown below.Status Standard Specification New Each MIHS unit covers a duration of haptic presentation time defined by a timestamp and a duration. Consecutive MIHS units shall either be temporally aligned, sharing the same timestamp and duration or be in sequence, i.e. a MIHS Unit starts at the end of the previous MIHS unit. The MIHS unit is followed by the next MIHS unit, unless it is the last MIHS unit of the haptic experience. All MIHS packets of a MIHS unit have the starting time and duration of the containing MIHS unit. An MIHS unit may have one of the following types:—Initialization. An initialization unit shall contain one timing MIHS packet and may includeone or more metadata MIHS packets. The initialization unit starts a new time anchor in haptics presentation time by setting the timestamp and may also set or change the timescale. The duration of an initialization unit duration shall be zero.—Temporal. A temporal unit shall contain one or more MIHS packets. The duration of atemporal unit shall be a positive number.—Spatial. A spatial unit shall contain one or more MIHS packets. The duration of a spatialunit shall be zero. If compatibility with the HJIF format is desired, all effects of a spatial modality need to be included in the first MIHS data unit.—Silent. A silent unit indicates that there is no effect that starts during this time interval andshall not include any MIHS packets. The duration of a silent unit shall be a positive number. The first MIHS unit in a haptic stream shall be an initialization unit. AMIHS unit may be a sync unit or a non-sync unit:—A sync unit resets the previous effects and therefore provides an independent hapticexperience from the previous MIHS units.—A non-sync unit is the continuation of previous MIHS units and cannot be independentlydecoded and rendered without decoding the previous MIHS unit(s). New If the last non-Spatial MIHS Unit before a temporal or a given Silent MIHS Unit is an Initialization MIHS Unit, the timestamp of this given MIHS Unit shall be equal to the timestamp of the timing packet in the MIHS Initialization Unit. If the last non-Spatial MIHS Unit before a second temporal or Silent MIHS Unit is a first temporal or silent MIHS Unit, the timestamp of the second temporal or silent MIHS Unit shall be equal to the timestamp of the first temporal or silent MIHS Unit or equal to the timestamp of the first temporal or silent MIHS Unit plus its duration. The duration of all MIHS Units with the same timestamp shall be equal.

[0132] An embodiment such as that of Example 1 may further be implemented using a modification to the syntax of the mpegiHapticUnit in the MPEG Haptic standard with the addition of a MIHSUnitTimestamp element as shown below. Syntax^ No.^ of^Mnemonic^bits^ mpegiHapticUnit() ^ ^{ ^ ^^ MIHSUnitType;^ 6^ uimsbf^ ^ MIHSUnitSync;^ 2^ uimsbf^ ^ MIHSUnitDuration;^ 24^ uimsbf^ ^ MIHSUnitTimestamp;^ 32^ uimsbf^ ^MIHSUnitLength;^ 32^ uimsbf^while (packetsAvalaible() != 0) { ^ ^mpegiHapticPacket(); ^ ^} ^ ^} ^ ^

[0133] The semantics of a MIHSUnitTimestamp element may be as follows.

[0134] MIHSUnitTimestamp: Timestamp of the MIHS unit. MIHSUnitTimestamp may be constrained to be equal to or greater than the timestamp of the readMetadataTiming() structure in the most recently sent mpegHapticUnit() with MIHSUnitType UNITTYPE_INITIALIZATION. The timestamp is in the timescale of the readMetadataTiming() structure in the most recently sent mpegHapticUnit() with MIHSUnitType UNITTYPE_INITIALIZATION. Example 2.

[0135] Example 2 is another implementation of an embodiment using additional timing information. In Example 2, the timing information is not added in the MIHS unit header but rather is added in an additional timing packet. In the current MPEG specification, timing packets can only be used in MIHS initialization units. In the example described here, a timing packet may be (or in some embodiments, must be) sent with each initialization, temporal and silent MIHS unit. An example implementation of Example 2 is shown schematically in FIG.14.

[0136] In implementations of Example 2, the haptics media encoder and decoder may be configured to ensure that all MIHS units are properly aligned and organized in MIHS Frames according to some or all of the following principles. ^ If the last non-spatial MIHS unit before a temporal or a silent MIHS unit is an initialization unit, the timestamp in the timing packet of this MIHS unit is constrained to be equal to the timestamp of the timing packet in the MIHS initialization unit. ^ If the last non-spatial MIHS unit before a second temporal or a silent MIHS unit is a first temporal or a silent MIHS unit, the timestamp in the timing packet of the second MIHS unit is constrained to be equal to the timestamp in the timing packet of the first temporal or silent MIHS unit or equal to the timestamp in the timing packet of the first temporal or silent MIHS unit plus the duration of the first temporal or silent MIHS unit. ^ The duration of all MIHS units with the same timestamp in the timing packet is constrained to be equal. ^ The timescale information in timing packets of temporal or silent MIHS units is constrained to be the same as the timescale information in timing packets of the previous initialization unit ^ Spatial units shall not contain timing packets.

[0137] An embodiment such as that of Example 2 may be implemented using a modification of the MPEG Haptic standard. For example, a requirement may be implemented such that all non- spatial MIHS unit contain a timing packet. In an example implementation, the MPEG Haptic standard is modified with the addition of new material as shown below.Status Standard Specification New Each MIHS unit covers a duration of haptic presentation time defined by a timestamp and a duration. Consecutive MIHS units shall either be temporally aligned, sharing the same timestamp and duration or be in sequence, i.e. a MIHS Unit starts at the end of the previous MIHS unit. The MIHS unit is followed by the next MIHS unit, unless it is the last MIHS unit of the haptic experience. All MIHS packets of a MIHS unit have the same start time and duration of the containing MIHS unit. An MIHS unit may have one of the following types:—Initialization. An initialization unit shall contain one timing MIHS packet and may includeone or more metadata MIHS packets. The initialization unit starts a new time anchor in haptics presentation time by setting the timestamp and may also set or change the timescale. The duration of an initialization unit duration shall be zero.—Temporal. A temporal unit shall contain one or more MIHS packets. The duration of atemporal unit shall be a positive number.—Spatial. A spatial unit shall contain one or more MIHS packets. The duration of a spatialunit shall be zero. If compatibility with the HJIF format is desired, all effects of a spatial modality need to be included in the first MIHS data unit.—Silent. A silent unit indicates that there is no effect that starts during this time interval. Theduration of a silent unit shall be a positive number. New A silent unit shall only include a single MIHS packets of type PACTYPE_TIMING. The first MIHS unit in a haptic stream shall be an initialization unit. AMIHS unit may be a sync unit or a non-sync unit:—A sync unit resets the previous effects and therefore provides an independent hapticexperience from the previous MIHS units.—A non-sync unit is the continuation of previous MIHS units and cannot be independentlydecoded and rendered without decoding the previous MIHS unit(s). New If the last non-Spatial MIHS Unit before a temporal or a Silent MIHS Unit is an initialization Unit, the timestamp in the timing packet of this MIHS Unit shall be equal to the timestamp of the timing packet in the MIHS Initialization Unit If the last non-Spatial MIHS Unit before second temporal or Silent MIHS Unit is a first temporal or silent MIHS Unit, the timestamp in the timing packet of the second temporal or silent MIHS Unit shall be equal to the timestamp in the timing packet of the first temporal or silent MIHS Unit or equal to the timestamp in the timing packet of the first temporal or silent MIHS Unit plus its duration. The duration of all MIHS Units with the same timestamp in the timing packet shall be equal. The timescale information in Timing Packets of temporal or Silent MIHS Units shall be the same as the timescale information in Timing Packets of the previous Initialization Unit Spatial Units shall not contain Timing packets. An MIHS unit comprises the unit header and MIHS packets as shown in FIG.21. Names in the figure are shortened with respect to their syntax counterparts for easier reading. Initialization units An initialization unit is a sync unit and shall have the following constraints:—Exactly one packet of type PACTYPE_TIMING— Zero or more packets of the following types:^ PACTYPE_METADATAEXPERIENCE ^ PACTYPE_METADATAPERCEPTION ^ PACTYPE_METADATACHANNEL ^ PACTYPE_METADATABAND ^PACTYPE_LIBRARYEFFECTS—Zero or more packets of the following types:^ PACTYPE_CRC16 ^ PACTYPE_CRC32 ^ PACTYPE_GlobalCRC16 ^PACTYPE_GlobalCRC32—No MIHS packet of the following types:^ PACTYPE_DATA The sync field shall be ignored for an initialization unit since it is a sync unit by definition.The duration field shall be ignored for an initialization unit since it does not have a duration by definition. Temporal and spatial units Temporal and spatial units shall have the following constraints:—One or more packets of the following types:^PACTYPE_DATA—Zero or more packets of the following types:^ PACTYPE_CRC16 ^ PACTYPE_CRC32 ^ PACTYPE_GlobalCRC16 ^PACTYPE_GlobalCRC32—No packets of following types:^ PACTYPE_METADATAEXPERIENCE ^ PACTYPE_METADATAPERCEPTION ^ PACTYPE_METADATACHANNEL ^ PACTYPE_METADATABAND ^ PACTYPE_LIBRARYEFFECTS Temporal units shall have the following additional constraint:—Only data for the following perception modalities:^ Pressure ^ Acceleration ^ Velocity ^ Position ^ Temperature ^ Vibrotactile ^ Water ^ Wind ^ Force ^ User-defined Temporal New Temporal units shall be constrained to have exactly one packet of type PACTYPE_TIMING. Spatial units shall have the following additional constraint:—Only data for the following perception modalities:^ Vibrotactile texture ^ Stiffness ^ Friction ^ User-defined SpatialNew — Spatial units shall be constrained to have no packet of type PACTYPE_TIMING.—Silent units shall have the following constraint:o Exactly one packet of type PACTYPE_TIMING A perception may include temporal or spatial units but not both. The sync field shall be ignored for a spatial unit since it is a sync unit by definition. The duration field shall be ignored for a spatial unit since it does not have a duration by definition.Embodiments using sequence numbers.

[0138] Some embodiments described herein make use of sequence numbers. In some examples of such embodiments, a sequence number is a reference to the last initialization MIHS unit, increasing with each frame. This sequence number can be interpreted, for example, as an id or a counter of MIHS frames. It provides an indication of which MIHS frame an MIHS unit belongs to. In example embodiments, every MIHS unit in the same MIHS frame shares the same sequence number and has the same duration.

[0139] Embodiments using sequence numbers allow the handling of congestion issues by allowing multiple MIHS units to share the same time span (and thus to be part of the same MIHS frame). Such embodiments, however, do not necessarily optimize the handling of packet loss. For example, if every packet in a given MIHS frame is lost, then the duration of the MIHS frame is unknown (similar to the problem illustrated in FIG.9) and the starting time of the MIHS units in the following MIHS frames cannot necessarily be inferred. However, some embodiments address this by using a duration of the MIHS frames that is constant throughout the bit-stream or in- between two Initialization MIHS unit, which allows the time stamp of an MIHS frame to be calculated using the sequence number and the Timing packet present in the MIHS initialization unit irrespective of the packet–loss scenarios. Alternatively or additionally, some embodiments address this issue through packet loss detection. If the receiver retrieves MIHS units and detect a gap in the sequence numbering, it operates in some embodiments to request the sender to resend the missing MIHS frames based on the missing sequence numbers. Example 3.

[0140] Example 3 is an implementation of an embodiment using sequence numbers. In Example 3, the sequence number is added using additional bits in the MIHS unit Header, as illustrated with the added sequence number property in FIG. 15. This example provides for temporal alignment of the MIHS units. In some embodiments, one or more of the following additional constraints are applied. ^ All MIHS units sharing the same sequence number between two initialization MIHS units or after the last Initialization MIHS unit are constrained to have the same duration. ^ The sequence number of initialization MIHS units is constrained to be zero. ^ The sequence number of silent and temporal MIHS units is constrained to be greater than zero. ^ The sequence number of a silent or temporal MIHS unit is constrained to be equal to the sequence number of the previous non-spatial MIHS unit or equal to the sequence number of the previous non-spatial MIHS unit plus one.

[0141] An embodiment such as that of Example 3 may be implemented using a modification of the MPEG Haptic standard. For example, some of the constraints defined in the specification maybe changed, and the decoder may be configured appropriately to read the header of the MIHS units as modified. In an example implementation, the MIHS clause of the MPEG Haptic standard is modified with the addition of new material as shown below. Status Standard Specification New Each MIHS unit covers a given duration of haptic presentation time. Consecutive MIHS units shall either be temporally aligned, covering the same time span or be in sequence, i.e. a MIHS Unit starts at the end of the previous MIHS unit. The time span covered by a MIHS Unit is defined by a duration and a sequence number. The sequence number indicates which time slot a MIHS Unit belongs to. MIHS Units covering the same time range shall have the same sequence number and all MIHS Units with the same sequence number shall have the same duration. The MIHS unit is followed by the next MIHS unit, unless it is the last MIHS unit of the haptic experience. All MIHS packets of a MIHS unit have the same start time and duration of the containing MIHS unit. An MIHS unit may have one of the following types:—Initialization. An initialization unit shall contain one timing MIHS packet and may includeone or more metadata MIHS packets. The initialization unit starts a new time anchor in haptics presentation time by setting the timestamp and may also set or change the timescale. The duration of an initialization unit duration shall be zero.—Temporal. A temporal unit shall contain one or more MIHS packets. The duration of atemporal unit shall be a positive number.—Spatial. A spatial unit shall contain one or more MIHS packets. The duration of a spatialunit shall be zero. If compatibility with the HJIF format is desired, all effects of a spatial modality need to be included in the first MIHS data unit.—Silent. A silent unit indicates that there is no effect that starts during this time interval andshall not include any MIHS packets. The duration of a silent unit shall be a positive number. The first MIHS unit in a haptic stream shall be an initialization unit. AMIHS unit may be a sync unit or a non-sync unit:—A sync unit resets the previous effects and therefore provides an independent hapticexperience from the previous MIHS units.—A non-sync unit is the continuation of previous MIHS units and cannot be independentlydecoded and rendered without decoding the previous MIHS unit(s). New All MIHS Units sharing the same sequence number between two initialization MIHS Units or after the last Initialization MIHS Unit shall have the same duration. The sequence number of initialization MIHS units shall be zero. The sequence number of silent and temporal MIHS units shall be greater than zero. The sequence number of a silent or temporal MIHS unit shall be equal to the sequence number of the previous non-Spatial MIHS Unit or be equal to the sequence number of the previous non-Spatial MIHS Unit plus one.

[0142] In some example implementations, the syntax of the mpegiHapticUnit in the MPEG Haptic standard is modified with the addition of the MIHSUnitSequenceNumber syntax element as shown below.Syntax^ No.^ of^Mnemonic^bits^ mpegiHapticUnit() ^ ^{ ^ ^^ MIHSUnitType;^ 6^ uimsbf^ ^ MIHSUnitSync;^ 2^ uimsbf^ ^ MIHSUnitDuration;^ 24^ uimsbf^ ^ MIHSUnitSequenceNumber;^ 24^ uimsbf^ ^MIHSUnitLength;^ 32^ uimsbf^while (packetsAvalaible() != 0) { ^ ^mpegiHapticPacket(); ^ ^} ^ ^} ^ ^

[0143] In some example implementations, the semantics of the MIHSUnitSequenceNumber syntax element may be as follows.

[0144] MIHSUnitSequenceNumber: Sequence Number of the MIHS unit. MIHSUnitSequenceNumber shall be equal to zero for MIHS Units with type UNITTYPE_INITIALIZATION. MIHSUnitSequenceNumber shall be equal to or be consecutive to the sequence number of in the most recently sent mpegHapticUnit() with MIHSUnitType different from UNITTYPE_SPATIAL. Example 4.

[0145] Example 4 is another implementation of an embodiment using sequence numbers. In Example 4, the sequence number information is not added in the MIHS unit header but rather in an additional sequence packet, as illustrated with the added sequence packet in FIG. 16. This example provides for temporal alignment of the MIHS units. In this example, one or more of the following additional constraints may be imposed: ^ All MIHS units sharing the same sequence number between two initialization MIHS units or after the last Initialization MIHS unit are constrained to have the same duration. ^ All non-spatial MIHS units are constrained to contain a sequence packet. ^ The sequence number of initialization MIHS units is constrained to be zero. ^ Spatial MIHS units shall not contain a sequence packet. ^ The sequence number of silent and temporal MIHS units is constrained to be greater than zero. ^ The sequence number of a silent or temporal MIHS unit is constrained to be equal to the sequence number of the previous non-spatial MIHS unit or equal to the sequence number of the previous non-spatial MIHS unit plus one.

[0146] An embodiment such as that of Example 4 may be implemented using a modification of the MPEG Haptic standard. For example, some of the constraints defined in the specification may be changed, and a new packet type may be added to the list of packets. The decoder may beconfigured appropriately to read the sequence packet. In an example implementation, the MIHS clause of the MPEG Haptic standard is modified with the addition of new material as shown below. Status Standard Specification New Each MIHS unit covers a given duration of haptic presentation time. Consecutive MIHS units shall either be temporally aligned, covering the same time span or be in sequence, i.e. a MIHS Unit starts at the end of the previous MIHS unit. The time span covered by a MIHS Unit is defined by a duration and a sequence number. The sequence number indicates which time slot a MIHS Unit belongs to. MIHS Unit covering the same time range shall have the same sequence number and all MIHS Unit with the same sequence number shall have the same duration. The MIHS unit is followed by the next MIHS unit, unless it is the last MIHS unit of the haptic experience. All MIHS packets of a MIHS unit have the starting time and duration of the containing MIHS unit. An MIHS unit may have one of the following types:—Initialization. An initialization unit shall contain one timing MIHS packet and may includeone or more metadata MIHS packets. The initialization unit starts a new time anchor in haptics presentation time by setting the timestamp and may also set or change the timescale. The duration of an initialization unit duration shall be zero.—Temporal. A temporal unit shall contain one or more MIHS packets. The duration of atemporal unit shall be a positive number.—Spatial. A spatial unit shall contain one or more MIHS packets. The duration of a spatialunit shall be zero. If compatibility with the HJIF format is desired, all effects of a spatial modality need to be included in the first MIHS data unit.—Silent. A silent unit indicates that there is no effect that starts during this time interval andshall not include any MIHS packets. The duration of a silent unit shall be a positive number. The first MIHS unit in a haptic stream shall be an initialization unit. AMIHS unit may be a sync unit or a non-sync unit:—A sync unit resets the previous effects and therefore provides an independent hapticexperience from the previous MIHS units.—A non-sync unit is the continuation of previous MIHS units and cannot be independentlydecoded and rendered without decoding the previous MIHS unit(s). New All MIHS Units sharing the same sequence number between two initialization MIHS Units or after the last Initialization MIHS Unit shall have the same duration All non-Spatial MIHS Units shall contain a Sequence packet The sequence number of initialization MIHS units shall be zero The sequence number of silent and temporal MIHS units shall be greater than zero Spatial MIHS Units shall not contain a Sequence packet The sequence number of silent and temporal MIHS units shall be greater than zero The sequence number of a silent or temporal MIHS unit shall be equal to the sequence number of the previous non-Spatial MIHS Unit or equal to the sequence number of the previous non-Spatial MIHS Unit plus one.

[0147] In some example implementations, the MIHS decoding section in the MPEG Haptic standard is modified. Some such embodiments use the existing mpegiHapticPacket() with the following syntaxSyntax^ No.^ of^Mnemonic^bits^ mpegiHapticPacket() ^ ^{ ^ ^^ MIHSPacketType;^ 6^ uimsbf^ ^ MIHSLayer;^ 4^ uimsbf^ ^ reserved;^ 5^ uimsbf^ ^MIHSPacketLength;^ 17^ uimsbf^MIHSPacketPayload(MIHSPacketType); ^ ^ByteAlignement^ 0‐7^ uimsbf^} ^ ^

[0148] In this syntax, the value MIHSPacketType indicates the payload type in the packet. The payload type may be one of the known values 0-10 or a new value 11 corresponding to a payload type PACTYPE_SEQUENCENUMBER, or a reserved value 12-15. MIHSPacketType^ Value^ PACTYPE_TIMING^ 0 PACTYPE_METADATAEXPERIENCE^ 1 PACTYPE_METADATAPERCEPTION^ 2 PACTYPE_METADATACHANNEL^ 3 PACTYPE_METADATABAND^ 4 PACTYPE_DATA^ 5 PACTYPE_LIBRARYEFFECTS^ 6 PACTYPE_CRC16^ 7 PACTYPE_CRC32^ 8 PACTYPE_GlobalCRC16^ 9 PACTYPE_GlobalCRC32^ 10 PACTYPE_SEQUENCENUMBER^ 11 / *^reserved^* / ^ 12–15

[0149] Decoders may skip packets with an unknown MIHSPacketType.

[0150] The semantics of other elements of mpegiHapticPacket() may be as follows: MIHSLayer^ Importance of the packet for scalability. Given a limited bandwidth,decoders may ignore packets with a high MIHSLayer layer value.reserved^ This value is reserved bits; decoders should ignore it. MIHSPacketLength^ Length of the payload in bytes.MIHSPacketPayload() Payload for the MIHS packet.ByteAlignement^ Padding with up to seven bits set to 0 for the MIHSPacket to be byte-aligned.

[0151] In some embodiments, the MIHSPacketPayload() structure has the following syntax, in which lines marked with a dagger (†) are newly added compared to existing syntax, introducing the syntax element readSequenceNumber(), which provides metadata related to the sequence number of a MIHS unit, in cases where the payload type is PACTYPE_SEQUENCENUMBER.Syntax^ No.^ of^Mnemonic^bits^ mpegiPacketPayload(MIHSPacketType) ^ ^{ ^ ^switch (MIHSPacketType) { ^ ^case PACTYPE_TIMING: ^ ^readMetadataTiming(); ^ ^break; ^ ^case PACTYPE_METADATAEXPERIENCE: ^ ^readMetadataExperience(); ^ ^break; ^ ^case PACTYPE_METADATAPERCEPTION: ^ ^readMetadataPerception(); ^ ^break; ^ ^case PACTYPE_METADATACHANNEL: ^ ^readMetadataChannel(); ^ ^break; ^ ^case PACTYPE_METADATABAND: ^ ^readMetadataBand(); ^ ^break; ^ ^case PACTYPE_DATA: ^ ^readData (); ^ ^break; ^ ^case PACTYPE_LIBRARYEFFECTS: ^ ^readLibrary(); ^ ^break; ^ ^case PACTYPE_CRC16: ^ ^case PACTYPE_CRC32: ^ ^case PACTYPE_GlobalCRC16: ^ ^case PACTYPE_GlobalCRC32: ^ ^readCRC(); ^ ^break; ^ ^case PACTYPE_ SEQUENCENUMBER: † ^ ^readSequenceNumber(); † ^ ^break; † ^ ^} ^ ^} ^ ^

[0152] The semantics may be as follows.readMetadataTiming() Metadata related to the timing of subsequent MIHS packets,described in subclause 7.2.5.readMetadataExperience() Metadata of the haptic experience being streamed, described insubclause 7.2.6.readMetadataPerception() Metadata of a perception of the haptic experience, described insubclause 7.2.8.readMetadataChannel() Metadata of a channel of the haptic experience, described insubclause 7.2.10.readMetadataBand() Metadata of a band of the haptic experience, described in subclause7.2.11.readData() Effects for a band of the haptic experience, described in subclause7.2.14.readLibrary() Predefined effects to be referenced from the bands of the hapticexperience, described in subclause 7.2.12.readCRC() Cyclic redundancy check value used to protect one or more directlyfollowing packets, described in subclause 0.readSequenceNumber() Metadata related to the sequence number of a MIHS Unit.

[0153] In some embodiments, the readSequenceNumber() structure has the following syntax.Syntax^ No.^of^Mnemonic^bits^ readMetadataTiming() ^ ^{ ^ ^^sequenceNumber;^ 32^ uimsbf^}^ ^

[0154] Within the readSequenceNumber() structure, the syntax element sequenceNumber provides the Sequence number of the MIHS unit. Embodiments using Start and End units.

[0155] Some embodiments described herein make use of two new types of MIHS units: frame start and frame end. In some examples of such embodiments, MIHS frames are delimited by bounding temporal MIHS units of the same frame within these two new types of MIHS unit. These new MIHS units type do not need to contain any packet. This approach addresses the congestion issues by defining the concept of MIHS frame. To further help solving the packet loss issue, this approach is combined in some cases with one or more of the previous embodiments, e.g. by adding a timing packet or a sequence packet to the frame start MIHS units. In some embodiments, start and end units are included only at the start and end of a frame that includes more than one MIHS unit, such that frames with only a single unit are not necessarily defined using start and end units.

[0156] Embodiments using start and end units allow for the grouping of the data for a given frame in multiple MIHS units, thus allowing a sender to handle congestion issues by reducing the number of MIHS units to send for a given frame. FIG.17 illustrates an example of how these new types of MIHS units can be used to define the frames. In this example for each frame, the MIHS data packets for the first channel and for the second channel are grouped in two distinct MIHS units. The MIHS units containing the haptic for each frame are bounded by an MIHS frame start MIHS unit and an MIHS Frame end MIHS unit. Example 5.

[0157] Example 5 is an implementation of an embodiment using start and end units. In this example, the following additional constraint may be imposed: ^ Every temporal MIHS unit bounded by the same frame start MIHS unit and frame end MIHS unit is constrained to have the same duration and it should be equal to the duration of the frame start MIHS unit.

[0158] An embodiment such as that of Example 5 may be implemented using a modification of the MPEG Haptic standard. For example, part of the MIHS overview may be changed by adding new MIHS unit types and multiple constraints in the specifications. The decoder may be configured appropriately to handle the start and end units. In an example implementation, the MIHS clause of the MPEG Haptic standard is modified with the addition of new material as shown below.Status Standard Specification New Each MIHS unit covers a given duration of haptic presentation time. Consecutive temporal MIHS units shall either be temporally aligned, covering the same time span or be in sequence, i.e. a MIHS Unit starts at the end of the previous MIHS unit. MIHS Unit sharing the same starting time and duration are bounded by FrameStart and FrameEnd MIHS units. The MIHS unit is followed by the next MIHS unit, unless it is the last MIHS unit of the haptic experience. All MIHS packets of a MIHS unit have the starting time and duration of the containing MIHS unit. An MIHS unit may have one of the following types:—Initialization. An initialization unit shall contain one timing MIHS packet and may includeone or more metadata MIHS packets. The initialization unit starts a new time anchor in haptics presentation time by setting the timestamp and may also set or change the timescale. The duration of an initialization unit duration shall be zero.—Temporal. A temporal unit shall contain one or more MIHS packets. The duration of atemporal unit shall be a positive number.—Spatial. A spatial unit shall contain one or more MIHS packets. The duration of a spatialunit shall be zero. If compatibility with the HJIF format is desired, all effects of a spatial modality need to be included in the first MIHS data unit.—Silent. A silent unit indicates that there is no effect that starts during this time interval andshall not include any MIHS packets. The duration of a silent unit shall be a positive number. New — FrameStart. A FrameStart unit indicates that all the following temporal Units until the next FrameEnd Unit cover the same time frame (i.e. they have the same starting time and duration). AFrameStart Unit shall not include any MIHS packet.—FrameEnd. A FrameEnd unit indicates that all the previous temporal Units since theprevious FrameStart Unit cover the same time frame (i.e. they have the same starting time and duration). A FrameEnd Unit shall not include any MIHS packet. The temporal or Silent MIHS Unit following a Frame End Unit starts at the end of the FrameEnd Unit. Only temporal MIHS Units shall be bounded by FrameStart and FrameEnd MIHS Units.

[0159] Some example implementations make use of the mpegiHapticUnit() data structure in the MIHS decoding section in the MPEG Haptic standard, which has the following syntax. Syntax^ No.^ of^Mnemonic^bits^ mpegiHapticUnit() ^ ^{ ^ ^^ MIHSUnitType;^ 6^ uimsbf^ ^ MIHSUnitSync;^ 2^ uimsbf^ ^ MIHSUnitDuration;^ 24^ uimsbf^ ^MIHSUnitLength;^ 32^ uimsbf^while (packetsAvalaible() != 0) { ^ ^mpegiHapticPacket(); ^ ^} ^ ^} ^ ^

[0160] Within that data structure, the syntax element MIHSUnitType identifies the type of MIHS unit. Its value may be one of the previously-defined values 0-3 or, according to example embodiments, its value may be a 4, representing a frame start unit, or a 5, representing a frame end unit. (In this and other examples in the present document, different values may be assigned to different semantics in different embodiments.) Decoders may skip units with an unknownMIHSUnitType. In some embodiments, the values of MIHSUnitType may correspond to thefollowing unit types.MIHSUnitType^ Value^ UNITTYPE_INITIALIZATION^ 0 UNITTYPE_TEMPORAL^ 1 UNITTYPE_SPATIAL^ 2 UNITTYPE_SILENT^ 3 UNITTYPE_FRAME_START^ 4 UNITTYPE_FRAME_END^ 5 / *^reserved^* / ^ 6–63Embodiments using constant frame duration.

[0161] In embodiments described above, the duration of MIHS frames (defined by the duration of the MIHS unit it contains) may vary from one frame to the next, adding flexibility to the format. In addition, in embodiments using sequence numbers or start / end units, where the starting time of an MIHS frame (and therefore the MIHS units it contains) is inferred from the starting time and duration of the previous frames, packet loss issues may still be a problem. As illustrated in FIG. 18, if all the MIHS units in an MIHS frame are lost (which is plausible since a frame may contain a single MIHS unit), then the timing information of the following frames will be lost. This may be partially addressed by detecting the loss of the packets (using the sequence numbers) and requesting the server to send the missing packet again, but it is not optimal and it does not help from a decoder standpoint.

[0162] To address this issue, some example embodiments make the duration of all MIHS frames (and therefore MIHS units) equal in between initialization MIHS units (or between the last initialization MIHS unit and the end of the file). This can be achieved, for example, by adding this constraint in the MPEG Haptic standard specification or by defining this duration directly in the initialization MIHS unit. In some embodiments, this is done by removing the duration property from the MIHS unit header and defining it in the timing packet of the initialization MIHS unit. FIG. 19 illustrates this proposal as a variation of Example 4, with the duration of frames being provided in the initialization MIHS unit. The use of equal durations may alternatively be implemented in combination with any other embodiment described in this document.

[0163] With the use of constant frame duration, the duration of the MIHS frames remains the same and therefore the starting time of a silent or temporal MIHS unit can be inferred by directly by the receiver from the sequence number using the timestamp and duration defined in the initialization MIHS unit. Such embodiments may be combined with other embodiments as described above to allow for handling of packet loss.

[0164] An embodiment using constant frame duration may be implemented using a modification of the MPEG Haptic standard. For example, part of the MIHS overview may be changed and a duration field may be added in the readMetadataTiming decoding function. In an example implementation, the MIHS clause of the MPEG Haptic standard is modified to read as follows, with the addition of constraint that the duration of an MIHS unit is defined by a duration field inthe timing packet of the previous Initialization MIHS Unit.Each MIHS unit covers a nonoverlapping duration of haptic presentation time, i.e., it starts at the endof the previous MIHS unit and covers the duration of time defined by the duration field in the timingpacket of the previous Initialization MIHS Unit. The MIHS unit is followed by the next MIHS unit,unless it is the last MIHS unit of the haptic experience. All MIHS packets of a MIHS unit have thestarting time and duration of the containing MIHS unit.

[0165] In some example implementations, the MIHS decoding section in the MPEG Haptic standard is modified with the removal of the duration information from the mpegiHapticUnit() data structure to the readMetadataTiming() data structure. With such an implementation, the syntax of mpegiHapticUnit() may be as follows. Syntax^ No.^of^Mnemonic^bits^ mpegiHapticUnit() ^ ^{ ^ ^^ MIHSUnitType;^ 6^ uimsbf^ ^ MIHSUnitSync;^ 2^ uimsbf^ ^ ^ ^ ^MIHSUnitLength;^ 32^ uimsbf^while (packetsAvalaible() != 0) { ^ ^mpegiHapticPacket(); ^ ^} ^ ^} ^ ^

[0166] In this example, the mpegiHapticUnit() structure does not include an MIHSUnitDuration element. Instead, the duration information is provided in a readMetadataTiming() data structure with the addition of a “duration” element as follows. Syntax^ No.^of^bits^ Mnemonic^readMetadataTiming() ^ ^{ ^ ^^ timestamp;^ 32^ uimsbf^ ^ timescale;^ 32^ uimsbf^ ^duration;^ 24^ uimsbf^^^

[0167] The semantics of a readMetadataTiming() data structure with the addition of a “duration” element may be as follows. timestamp^ Timestamp of the haptic experience in ticks, i.e., the timestamp in seconds^is^timestamp / timescale.timescale^Number of ticks per second. The default value is 1000.duration^ Duration of all MIHS Units following the Initialization MIHS Unit containing thistiming packetEmbodiments using Layer / Priority information.

[0168] Example embodiments as described above provide different ways of organizing multiple MIHS units over the same time range. This allows for the handling of congestion issues on the network as it allows the server / streamer to limit the number and size of packets it has to send for a given time range. This organization of the units, however, is not sufficient to indicate which packets should be prioritized. This information is application dependent. In some cases, it might be useful to send only data related to a limited number of channels to provide haptic feedback on a reduced number of haptic actuators. For other applications it might be beneficial to reduce the quality of the stream by not sending the data packets containing the high frequency bands. Packet selection may also be based on the modality of the different perceptions. For example, it may be desirable in some instances to send only vibrotactile data and not kinesthetic data.

[0169] In order to allow the streamer to select the proper packets, example embodiments are provided that add priority information to the MIHS unit. For example, some embodiments include a layer property provided in each MIHS unit to indicate the level of priority of the MIHS unit. In some embodiments, the lower the value, the more important the MIHS unit. This information may be added directly in the MIHS unit header as illustrated in FIG.20, for example, or by providing a new packet type containing this information (a layer packet for instance), or using other techniques. The embodiment illustrated in FIG.20 is a variation of Example 4 described above. The use of priority information may be combined with any of the embodiments described herein. In addition, this added layer property of MIHS units may be used for other purposes, such as scalability.

[0170] Some embodiments include the constraint that the layer value of an initialization MIHS unit is constrained to be 0 (or other value as appropriate to indicate the highest priority level).

[0171] An embodiment using layer / priority information may be implemented using a modification of the MPEG Haptic standard. For example, an MIHSLayer field may be added in the readMetadataTiming decoding function. In an example implementation, the MIHS decoding section is modified to add priority information to the MIHS unit header with the addition of an MIHSLayer syntax element, resulting in a syntax as shown below. ^ Syntax^ No.^of^Mnemonic^bits^ mpegiHapticUnit() ^ ^{ ^ ^^ MIHSUnitType;^ 6^ uimsbf^ ^ MIHSUnitSync;^ 2^ uimsbf^ ^ MIHSUnitDuration;^ 24^ uimsbf^ ^^^^^^^MIHSLayer^ 4^ uimsbf^ ^MIHSUnitLength;^ 32^ uimsbf^while (packetsAvalaible() != 0) { ^ ^mpegiHapticPacket(); ^ ^} ^ ^} ^ ^^

[0172] The semantics of the MIHSLayer syntax element may be as follows. MIHSLayer^ Priority of MIHS Units for scalability. The lower the value the more important theMIHS Unit.

[0173] The implementation details provided herein for the different embodiments focus on the MIHS clause of the MPEG haptic standard specification. In order to minimize the changes to the current MPEG Haptic standard, in some implementations, MIHS Frames / Samples are organized based on data provided within the MIHS unit and packet data structures. Such implementations do not require the definition of a separate Frame data structure. However, in alternative embodiments, a new Frame data structure may be provided to encapsulate MIHS units. Further Embodiments.

[0174] A haptic decoding method according to some embodiments comprises: obtaining an encoded haptic signal including a plurality of haptic streaming units; obtaining frame information arranging the haptic streaming units in respective frames, the frame information including a respective timing packet in each of a plurality of the haptic streaming units; for at least a plurality of the haptic streaming units, determining a temporal period associated with the respective haptic streaming unit based on the timing packets; and decoding at least a first band of the encoded haptic signal from a plurality of first-band data packets, each of the first-band data packets being obtained from a respective one of the haptic streaming units.

[0175] A haptic decoding apparatus according to some embodiments comprises one or more processors configured to perform at least: obtaining an encoded haptic signal including a plurality of haptic streaming units; obtaining frame information arranging the haptic streaming units in respective frames, the frame information including a respective timing packet in each of a plurality of the haptic streaming units; for at least a plurality of the haptic streaming units, determining a temporal period associated with the respective haptic streaming unit based on the timing packets; and decoding at least a first band of the encoded haptic signal from a plurality of first-band data packets, each of the first-band data packets being obtained from a respective one of the haptic streaming units.

[0176] A haptic encoding method according to some embodiments comprises: obtaining information describing a haptic signal having at least a first band; encoding the haptic signal in a plurality of haptic streaming units, at least some of the haptic streaming units being associated with a respective temporal period, wherein encoding the haptic signal includes encoding the first band of the haptic signal as a plurality of first-band data packets and storing each of the first-banddata packets in a respective one of the haptic streaming units; and encoding frame information arranging the haptic streaming units in respective frames such that all haptic streaming units in a same frame are associated with the same temporal period, the frame information including a respective timing packet in each of a plurality of the haptic streaming units.

[0177] A haptic encoding apparatus according to some embodiments comprises one or more processors configured to perform at least: obtaining information describing a haptic signal having at least a first band; encoding the haptic signal in a plurality of haptic streaming units, at least some of the haptic streaming units being associated with a respective temporal period, wherein encoding the haptic signal includes encoding the first band of the haptic signal as a plurality of first-band data packets and storing each of the first-band data packets in a respective one of the haptic streaming units; and encoding frame information arranging the haptic streaming units in respective frames such that all haptic streaming units in a same frame are associated with the same temporal period, the frame information including a respective timing packet in each of a plurality of the haptic streaming units.

[0178] A method according to some embodiments comprises: obtaining information describing a haptic signal having at least a first band; encoding the haptic signal in a plurality of haptic streaming units, at least some of the haptic streaming units being associated with respective temporal period, wherein encoding the haptic signal includes encoding the first band of the haptic signal as a plurality of first-band data packets and storing each of the first-band data packets in a respective one of the haptic streaming units; and encoding frame information arranging the haptic streaming units in respective frames such that all haptic streaming units in a same frame are associated with the same temporal period.

[0179] In some embodiments, the frame information includes one or more of the following: time stamp information provided in a header of respective haptic streaming units; time stamp information provided in a timing packet in respective haptic streaming units; sequence information provided in a header of respective haptic streaming units; sequence information provided in a timing packet in respective haptic streaming units; at least one start packet indicating a start of a frame; and / or at least one end packet indicating an end of a frame.

[0180] In some embodiments, at least one of the frames includes at least two haptic streaming units.

[0181] In some embodiments, all of the frames are associated with a constant frame duration.

[0182] In some embodiments, each of the haptic streaming units includes layer information indicating a priority level of the respective haptic streaming unit. Some such embodiments further include: streaming the encoded haptics signal; detecting network congestion; and in response to the network congestion, selecting at least one haptic streaming unit to be removed from the stream based at least in part on the layer information. In some embodiments, the detecting of congestion and streaming of the encoded haptics signal is performed by a different entity thanthe entity that encodes the haptics signal. For example the entity that streams the haptics signal may obtain the encoded haptics signal without actually performing the encoding itself.

[0183] A method according to some embodiments comprises: obtaining information describing a haptic signal having at least a first band; and encoding the haptic signal in a plurality of haptic streaming units, at least some of the haptic streaming units including time stamp information indicating a start time of a temporal period associated with the respective haptic streaming unit; wherein encoding the haptic signal includes encoding the first band of the haptic signal as a plurality of first-band data packets and storing each of the first-band data packets in a respective one of the haptic streaming units.

[0184] In some embodiments, at least two of the haptic streaming units include the same time stamp information.

[0185] In some embodiments, a plurality of the haptic streaming units include duration information representing a duration of the respective temporal period, and all haptic streaming units with the same time stamp information also have the same duration information.

[0186] In some embodiments, the time stamp information is provided in a header of the respective haptic streaming unit.

[0187] In some embodiments, the time stamp information is provided in a header of the respective haptic streaming unit.

[0188] In some embodiments, the time stamp information is provided in a timing packet within the respective haptic streaming unit.

[0189] In some embodiments, the periods of time associated with the haptic streaming units have the same duration.

[0190] In some embodiments, each of the haptic streaming units includes layer information indicating a priority level of the respective haptic streaming unit.

[0191] A method according to some embodiments comprises: obtaining information describing a haptic signal having at least a first band; and encoding the haptic signal in a plurality of haptic streaming units, at least some of the haptic streaming units including sequence information indicating a position of the respective haptic streaming unit in a sequence of haptic streaming units; wherein encoding the haptic signal includes encoding the first band of the haptic signal as a plurality of first-band data packets and storing each of the first-band data packets in a respective one of the haptic streaming units.

[0192] In some embodiments, at least two of the haptic streaming units include the same sequence information.

[0193] In some embodiments, a plurality of the haptic streaming units include duration information representing a duration of the respective temporal period, and wherein all haptic streaming units with the same sequence information also have the same duration information.

[0194] In some embodiments, the sequence information is provided in a header of the respective haptic streaming unit.

[0195] In some embodiments, the sequence information is provided in a header of the respective haptic streaming unit.

[0196] In some embodiments, the sequence information is provided in a sequence packet within the respective haptic streaming unit.

[0197] In some embodiments, the periods of time associated with the haptic streaming units have the same duration.

[0198] In some embodiments, each of the haptic streaming units includes layer information indicating a priority level of the respective haptic streaming unit.

[0199] A method according to some embodiments comprises: obtaining information describing a haptic signal having at least a first band; and encoding the haptic signal in a plurality of haptic streaming units, at least some of the haptic streaming units being associated with a respective temporal period; wherein encoding the haptic signal includes encoding the first band of the haptic signal as a plurality of first-band data packets and storing each of the first-band data packets in a respective one of the haptic streaming units; and wherein, for each temporal period associated with one or more haptic streaming units, a start unit is provided before the haptic streaming units associated with that temporal period and an end unit is provided after the haptic streaming units associated with that temporal period.

[0200] In some embodiments, for at least one temporal period, at least two of the haptic streaming units are between the start unit and the end unit.

[0201] In some embodiments, a plurality of the haptic streaming units include duration information representing a duration of the respective temporal period, and wherein all haptic streaming units between the same start unit and end unit have the same duration information.

[0202] In some embodiments, information indicating the duration of the temporal period is provided in the respective start unit.

[0203] In some embodiments, the periods of time associated with the haptic streaming units have the same duration.

[0204] In some embodiments, each of the haptic streaming units includes layer information indicating a priority level of the respective haptic streaming unit.

[0205] A method according to some embodiments comprises: obtaining an encoded haptic signal including a plurality of haptic streaming units; obtaining frame information arranging the haptic streaming units in respective frames; for at least a plurality of the haptic streaming units, determining a temporal period associated with the respective haptic streaming unit based at least in part on the frame information; and decoding at least a first band of the encoded haptic signalfrom a plurality of first-band data packets, each of the first-band data packets being obtained from a respective one of the haptic streaming units.

[0206] Some embodiments further comprise rendering at least the first band of the encoded haptic signal.

[0207] In some embodiments, at least one of the frames includes at least two haptic streaming units.

[0208] Some embodiments further comprise decoding at least a second band of the encoded haptic signal from a plurality of second-band data packets, each of the second-band data packets being obtained from a respective one of the haptic streaming units.

[0209] In some embodiments, at least one of the frames includes a first haptic streaming unit including one of the first-band data packets and a second haptic streaming unit including one of the second-band data packets.

[0210] In some embodiments, the frame information includes one or more of the following: time stamp information provided in a header of respective haptic streaming units; time stamp information provided in a timing packet in respective haptic streaming units; sequence information provided in a header of respective haptic streaming units; sequence information provided in a timing packet in respective haptic streaming units; at least one start packet indicating a start of a frame; and / or at least one end packet indicating an end of a frame.

[0211] In some embodiments, all of the haptic streaming units in a frame have the same start time and the same duration.

[0212] Some embodiments include an apparatus comprising one or more processors configured to perform any of the methods described herein.

[0213] An apparatus according to some embodiments comprises a computer-readable medium storing instructions for causing one or more processors to perform any of the methods described herein.

[0214] An apparatus comprising at least one processor and at least one non-transitory computer-readable medium storing instructions for causing the at least one processor to perform any of the methods described herein.

[0215] A computer-readable medium according to some embodiments stores a bitstream generated according to any of the methods described herein.

[0216] Some embodiments include a signal including a bitstream generated according to any one of the methods described herein.

[0217] This disclosure describes a variety of aspects, including tools, features, embodiments, models, approaches, etc. Many of these aspects are described with specificity and, at least to show the individual characteristics, are often described in a manner that may sound limiting. However, this is for purposes of clarity in description, and does not limit the disclosure or scopeof those aspects. Indeed, all of the different aspects can be combined and interchanged to provide further aspects. Moreover, the aspects can be combined and interchanged with aspects described in earlier filings as well.

[0218] The aspects described and contemplated in this disclosure can be implemented in many different forms. While some embodiments are illustrated specifically, other embodiments are contemplated, and the discussion of particular embodiments does not limit the breadth of the implementations. At least one of the aspects generally relates to encoding and decoding, and at least one other aspect generally relates to transmitting a bitstream generated or encoded. These and other aspects can be implemented as a method, an apparatus, a computer readable storage medium having stored thereon instructions for encoding or decoding data according to any of the methods described, and / or a computer readable storage medium having stored thereon a bitstream generated according to any of the methods described.

[0219] Various methods are described herein, and each of the methods comprises one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as “first”, “second”, etc. may be used in various embodiments to modify an element, component, step, operation, etc., such as, for example, a “first decoding” and a “second decoding”. Use of such terms does not imply an ordering to the modified operations unless specifically required. So, in this example, the first decoding need not be performed before the second decoding, and may occur, for example, before, during, or in an overlapping time period with the second decoding.

[0220] Various numeric values may be used in the present disclosure, for example. The specific values are for example purposes and the aspects described are not limited to these specific values.

[0221] Embodiments described herein may be carried out by computer software implemented by a processor or other hardware, or by a combination of hardware and software. As a non- limiting example, the embodiments can be implemented by one or more integrated circuits. The processor can be of any type appropriate to the technical environment and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as non-limiting examples.

[0222] Various implementations involve decoding. “Decoding”, as used in this disclosure, can encompass all or part of the processes performed, for example, on a received encoded sequence in order to produce a final output suitable for presentation. In various embodiments, such processes include one or more of the processes typically performed by a decoder, for example, entropy decoding, inverse quantization, inverse transformation, and differential decoding. In various embodiments, such processes also, or alternatively, include processes performed by a decoder of various implementations described in this disclosure.

[0223] Various implementations involve encoding. In an analogous way to the above discussion about “decoding”, “encoding” as used in this disclosure can encompass all or part of the processes performed, for example, on an input in order to produce an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, for example, partitioning, differential encoding, transformation, quantization, and entropy encoding. In various embodiments, such processes also, or alternatively, include processes performed by an encoder of various implementations described in this disclosure.

[0224] When a figure is presented as a flow diagram, it should be understood that it also provides a block diagram of a corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow diagram of a corresponding method / process.

[0225] The implementations and aspects described herein can be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed can also be implemented in other forms (for example, an apparatus or program). An apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end- users.

[0226] Reference to “one embodiment” or “an embodiment” or “one implementation” or “an implementation”, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment” or “in one implementation” or “in an implementation”, as well any other variations, appearing in various places throughout this disclosure are not necessarily all referring to the same embodiment.

[0227] Additionally, this disclosure may refer to “determining” various pieces of information. Determining the information can include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

[0228] Further, this disclosure may refer to “accessing” various pieces of information. Accessing the information can include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0229] Additionally, this disclosure may refer to “receiving” various pieces of information. Receiving is, as with “accessing”, intended to be a broad term. Receiving the information can include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further, “receiving” is typically involved, in one way or another, during operations such as, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0230] It is to be appreciated that the use of any of the following “ / ”, “and / or”, and “at least one of”, for example, in the cases of “A / B”, “A and / or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended for as many items as are listed.

[0231] Also, as used herein, the word “signal” refers to, among other things, indicating something to a corresponding decoder. For example, in certain embodiments the encoder signals a particular one of a plurality of parameters for region-based filter parameter selection for de- artifact filtering. In this way, in an embodiment the same parameter is used at both the encoder side and the decoder side. Thus, for example, an encoder can transmit (explicit signaling) a particular parameter to the decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter as well as others, then signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding transmission of any actual functions, a bit savings is realized in various embodiments. It is to be appreciated that signaling can be accomplished in a variety of ways. For example, one or more syntax elements, flags, and so forth are used to signal information to a corresponding decoder in various embodiments. While the preceding relates to the verb form of the word “signal”, the word “signal” can also be used herein as a noun.

[0232] Implementations can produce a variety of signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal can be formatted to carry the bitstream of a described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that thesignal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0233] We describe a number of embodiments. Features of these embodiments can be provided alone or in any combination, across various claim categories and types.

[0234] Note that various hardware elements of one or more of the described embodiments are referred to as “modules” that carry out (i.e., perform, execute, and the like) various functions that are described herein in connection with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices) deemed suitable for a given implementation. Each described module may also include instructions executable for carrying out the one or more functions described as being carried out by the respective module, and it is noted that those instructions could take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, software instructions, and / or the like, and may be stored in any suitable non-transitory computer-readable medium or media, such as commonly referred to as RAM, ROM, etc.

[0235] Although features and elements are described above in particular combinations, each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto- optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

CLAIMS 1. A haptic decoding method comprising: obtaining an encoded haptic signal including a plurality of haptic streaming units; obtaining frame information arranging the haptic streaming units in respective frames, the frame information including a respective timing packet in each of a plurality of the haptic streaming units; for at least a plurality of the haptic streaming units, determining a temporal period associated with the respective haptic streaming unit based on the timing packets; and decoding at least a first band of the encoded haptic signal from a plurality of first-band data packets, each of the first-band data packets being obtained from a respective one of the haptic streaming units.

2. A haptic decoding apparatus comprising one or more processors configured to perform at least: obtaining an encoded haptic signal including a plurality of haptic streaming units; obtaining frame information arranging the haptic streaming units in respective frames, the frame information including a respective timing packet in each of a plurality of the haptic streaming units; for at least a plurality of the haptic streaming units, determining a temporal period associated with the respective haptic streaming unit based on the timing packets; and decoding at least a first band of the encoded haptic signal from a plurality of first-band data packets, each of the first-band data packets being obtained from a respective one of the haptic streaming units.

3. The method of claim 1 or the apparatus of claim 2, further comprising rendering at least the first band of the encoded haptic signal.

4. The method of claim 1 or claim 3 as it depends from claim 1, or the apparatus of claim 2 or claim 3 as it depends from claim 2, wherein at least one of the frames includes at least two haptic streaming units.

5. The method of claim 1 or claims 3-4 as they depend from claim 1, or the apparatus of claim 2 or claims 3-4 as they depend from claim 2, further comprising decoding at least a second band of the encoded haptic signal from a plurality of second-band data packets, each of the second-band data packets being obtained from a respective one of the haptic streaming units.

6. The method of claim 1 or claims 3-5 as they depend from claim 1, or the apparatus of claim 2 or claims 3-5 as they depend from claim 2, wherein at least one of the frames includes a firsthaptic streaming unit including one of the first-band data packets and a second haptic streaming unit including one of the second-band data packets.

7. The method of claim 1 or claims 3-6 as they depend from claim 1, or the apparatus of claim 2 or claims 3-6 as they depend from claim 2, wherein each timing packet includes a time stamp and a duration.

8. The method of claim 1 or claims 3-6 as they depend from claim 1, or the apparatus of claim 2 or claims 3-6 as they depend from claim 2, wherein each timing packet includes a time stamp and a duration, wherein at least one frame includes a plurality of haptic streaming units, and wherein all haptic streaming units in the frame have a timing packet with a same time stamp and a same duration.

9. The method of claim 1 or claims 3-8 as they depend from claim 1, or the apparatus of claim 2 or claims 3-8 as they depend from claim 2, wherein each of the haptic streaming units includes layer information indicating a priority level of the respective haptic streaming unit.

10. A haptic encoding method comprising: obtaining information describing a haptic signal having at least a first band; encoding the haptic signal in a plurality of haptic streaming units, at least some of the haptic streaming units being associated with a respective temporal period, wherein encoding the haptic signal includes encoding the first band of the haptic signal as a plurality of first- band data packets and storing each of the first-band data packets in a respective one of the haptic streaming units; and encoding frame information arranging the haptic streaming units in respective frames such that all haptic streaming units in a same frame are associated with the same temporal period, the frame information including a respective timing packet in each of a plurality of the haptic streaming units.

11. A haptic encoding apparatus comprising one or more processors configured to perform at least: obtaining information describing a haptic signal having at least a first band; encoding the haptic signal in a plurality of haptic streaming units, at least some of the haptic streaming units being associated with a respective temporal period, wherein encoding the haptic signal includes encoding the first band of the haptic signal as a plurality of first- band data packets and storing each of the first-band data packets in a respective one of the haptic streaming units; and encoding frame information arranging the haptic streaming units in respective frames such that all haptic streaming units in a same frame are associated with the same temporalperiod, the frame information including a respective timing packet in each of a plurality of the haptic streaming units.

12. The method of claim 10 or the apparatus of claim 11, wherein each timing packet includes a time stamp and a duration, wherein at least one frame includes a plurality of haptic streaming units, and wherein all haptic streaming units in the frame have a timing packet with a same time stamp and a same duration.

13. The method of claim 10 or claim 12 as it depends from claim 10, or the apparatus of claim 11 or claim 12 as it depends from claim 11, wherein encoding the haptic signal includes encoding a second band of the haptic signal as a plurality of second-band data packets and storing each of the second-band data packets in a respective one of the haptic streaming units.

14. The method of claim 10 or claims 12-13 as they depend from claim 10, or the apparatus of claim 11 or claims 12-13 as they depend from claim 11, wherein each of the haptic streaming units includes layer information indicating a priority level of the respective haptic streaming unit.

15. The method of claim 10 or claims 12-13 as they depend from claim 10, or the apparatus of claim 11 or claims 12-13 as they depend from claim 11, wherein each of the haptic streaming units includes layer information indicating a priority level of the respective haptic streaming unit, further comprising: streaming the encoded haptics signal; detecting network congestion; and in response to the network congestion, selecting at least one haptic streaming unit to be removed from the stream based at least in part on the layer information.

Images