Qos-aware haptic data transmission methods

Abstract

Systems, methods, and instrumentalities are disclosed for performing haptic data transmission. A first haptic device may determine the set of parameters based on at least one of an application logic or information associated with a second haptic device. The first haptic device may send an offer message to the second haptic device. The offer message may include the set of parameters associated with the plurality of portions of a haptic media bitstream. The first haptic device may receive an answer message from the second haptic receiving device. The answer message may include indication(s) of a selection of a subset of parameters from the set of parameters initially offered by the first haptic device. The first haptic device, based on the selection of the parameters may select a portion of the haptic media bitstream, and send the selected portion of the haptic media bitstream to the second haptic device.

Description

QOS-AWARE HAPTIC DATA TRANSMISSION METHODSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of US Provisional Patent Application No. 63 / 544,009, filed October 13, 2023, the disclosure of which is incorporated herein by reference in its entirety.BACKGROUND

[0002] Video coding systems may be used to compress digital video signals, e.g., to reduce the storage and / or transmission bandwidth needed for such signals. Video coding systems may include, for example, block-based, wavelet-based, and / or object-based systems.SUMMARY

[0003] Systems, methods, and instrumentalities are disclosed for haptic data transmission. A first haptic device (e.g., a haptic sending device) may determine a set of methods and / or parameters associated with a plurality of a haptic media bitstream. The haptic media bitstream comprises a haptic media packet, wherein the haptic media packet may be marked with one parameter of the parameter set, and wherein the parameter enables the haptic receiving device or an intermediate device to apply a differentiated service. Each portion of the plurality of the portions of the haptic media bitstream may be associated with a differentiated quality of service (QoS).

[0004] The first haptic device may determine the set of parameters based on at least one of an application logic or information associated with a second haptic device (e.g., a haptic receiving device). The first haptic device may send an offer message to the second haptic device. The set of parameters may indicate at least one of a range of available band frequencies, a level of details to be used for avatars, a set of haptic perceptions represented by one or more avatars, a location of one or more devices or actuators on a body, a maximum frequency value of an actuator, a minimum frequency value of the actuator, an actuator type, or a set of haptic perception modalities.

[0005] The offer message may include the set of parameters associated with the plurality of portions of a haptic media bitstream. The first haptic device may receive an answer message from the second haptic receiving device. The answer message may include indication(s) of a selection of a subset of parameters from the set of parameters initially offered by the first haptic device. In examples, the offer message or the answer message may be communicated via a per-packet haptics transport information element (IE). The per-packet haptics transport IE is at least one of a sync type IE, a unit type IE, a dependency stamp IE, a dependency identifier (ID) IE, a haptic transport unit (HTU) layer IE, a transmission type IE, or a haptic sequence number IE. In examples, the first haptic device may determine a priority value associated with an HTU, and wherein the HTU layer IE indicates a priority level associated with the HTU.

[0006] The per-packet haptics transport IE may include indication of at least one of the following: a sync type indicating whether a MPEG-I Haptic Streaming (MIHS) unit is a sync packet; a unit type indicating a priority for the per-packet haptics transport IE; or a MIHS layer indicating a priority order of the MIHS unit. The transmission type IE indicates transmission type of a haptic media packet, and wherein the transmission type is of a single MPEG-I Haptic Streaming (MIHS) packet mode or a fragmented MIHS packet mode.

[0007] The per-flow haptics transport IE comprises at least one of the following: a frequency range IE, a perception modality IE, a type of actuator IE, a layer information IE, a haptics-encoding-method IE, a body part mask IE, a haptic silence suppression IE, a default haptics signal IE, a version IE, a profile IE, a level IE, a layer vs. sync priority IE, a media-type-priority IE, a haptics congestion control configuration IE, or a selective haptics loss recovery configuration IE.

[0008] The first haptic device, based on the selection of the parameters may select a portion of the haptic media bitstream. The first haptic device may then send the selected portion of the haptic media bitstream to the second haptic device.

[0009] Systems, methods, and instrumentalities are disclosed for performing haptic data transmission.

[0010] A device (e.g., a sender device) may determine one or more available methods that may be used to identify a quality of service (QoS) that may be used to transmit a portion of a haptic stream. The device may send an offer message that includes an indication of one or more available methods. The device may receive an answer message that includes an indication of a selected method from the one or more available methods. The device may select haptics data using the selected method. The device may send the selected haptics data.

[0011] The device may determine, for each method of the one or more available methods, one or more related parameters associated with the respective method. In examples, at least one of the one or more related parameters may be a range of available band frequencies in the haptics stream.

[0012] The offer message may include an indication of one or more related parameters for each of the one or more available methods. The answer message may include an indication of a selected parameter from the one or more related parameters associated with the selected method. The haptic data may be selected using the selected parameter. The offer message may include one or more per-flow haptics transport information elements (lEs). The answer message may include one or more per-flow haptics transport information elements (lEs).

[0013] Systems, methods, and instrumentalities described herein may involve a decoder. In some examples, the systems, methods, and instrumentalities described herein may involve an encoder. In some examples, the systems, methods, and instrumentalities described herein may involve a signal (e.g., from an encoder and / or received by a decoder). A computer-readable medium may include instructions for causing one or more processors to perform methods described herein. A computer program product may include instructions which, when the program is executed by one or more processors, may cause the one or more processors to carry out the methods described herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG. 1 A is a system diagram illustrating an example communications system in which one or more disclosed embodiments may be implemented.

[0015] FIG. 1 B is a system diagram illustrating an example wireless transmit / receive unit (WTRU) that may be used within the communications system illustrated in FIG. 1 A according to an embodiment.

[0016] FIG. 1 C is a system diagram illustrating an example radio access network (RAN) and an example core network (CN) that may be used within the communications system illustrated in FIG. 1 A according to an embodiment.

[0017] FIG. 1 D is a system diagram illustrating a further example RAN and a further example CN that may be used within the communications system illustrated in FIG. 1 A according to an embodiment.

[0018] FIG. 2 illustrates an example video encoder.

[0019] FIG. 3 illustrates an example video decoder.

[0020] FIG. 4 illustrates an example of a a system in which various aspects and examples may be implemented.

[0021] FIG. 5 illustrates an example of a system for efficiently transmitting haptic data.

[0022] FIG. 6 illustrates example transmission modes in real-time transport protocol (RTP).

[0023] FIGS. 7A and 7B illustrate an example haptics traffic negotiation.

[0024] FIG. 8 illustrates an example of network congestion control.

[0025] FIG. 9 illustrates an example of a dependency stamp.

[0026] FIG. 10 illustrates an example of packet loss recovery.DETAILED DESCRIPTION

[0027] A more detailed understanding may be had from the following description, given by way of example in conjunction with the accompanying drawings.

[0028] FIG. 1 A is a diagram illustrating an example communications system 100 in which one or more disclosed embodiments may be implemented. The communications system 100 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communications system 100 may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communications systems 100 may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block-filtered OFDM, filter bank multicarrier (FBMC), and the like.

[0029] As shown in FIG. 1A, the communications system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, a RAN 104 / 113, a ON 106 / 115, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, though it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d, any of which may be referred to as a “station” and / or a “STA”, may be configured to transmit and / or receive wireless signals and may include a user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smartphone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (loT) device, a watch or other wearable, a head-mounted display (HMD), a vehicle, a drone, a medical device and applications (e.g., remote surgery), an industrial device and applications (e.g., a robot and / or other wireless devices operating in an industrial and / oran automated processing chain contexts), a consumer electronics device, a device operating on commercial and / or industrial wireless networks, and the like. Any of the WTRUs 102a, 102b, 102c and 102d may be interchangeably referred to as a UE.

[0030] The communications systems 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN 106 / 115, the Internet 110, and / or the other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a gNB, a NR NodeB, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each depicted as a single element, it will be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0031] The base station 114a may be part of the RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and / or the base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage for a wireless service to a specific geographical area that may be relatively fixed or that may change over time. The cell may further be divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e., one for each sector of the cell. In an embodiment, the base station 114a may employ multiple-input multiple output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in desired spatial directions.

[0032] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0033] More specifically, as noted above, the communications system 100 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC- FDMA, and the like. For example, the base station 114a in the RAN 104 / 113 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) TerrestrialRadio Access (UTRA), which may establish the air interface 115 / 116 / 117 using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High-Speed Downlink (DL) Packet Access (HSDPA) and / or High-Speed UL Packet Access (HSUPA).

[0034] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro).

[0035] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as NR Radio Access, which may establish the air interface 116 using New Radio (NR).

[0036] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may implement LTE radio access and NR radio access together, for instance using dual connectivity (DC) principles. Thus, the air interface utilized by WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., a eNB and a gNB).

[0037] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS- 2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and the like.

[0038] The base station 114b in FIG. 1 A may be a wireless router, Home Node B, Home eNode B, or access point, for example, and may utilize any suitable RAT for facilitating wireless connectivity in a localized area, such as a place of business, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a roadway, and the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In an embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR etc.) to establish a picocell or femtocell. As shown in FIG. 1A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not be required to access the Internet 110 via the CN 106 / 115.

[0039] The RAN 104 / 113 may be in communication with the CN 106 / 115, which may be any type of network configured to provide voice, data, applications, and / or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have varying quality of service (QoS) requirements, such as differing throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, and the like. The CN 106 / 115 may provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, etc., and / or perform high-level security functions, such as user authentication. Although not shown in FIG. 1A, it will be appreciated that the RAN 104 / 113 and / or the CN 106 / 115 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 104 / 113 or a different RAT. For example, in addition to being connected to the RAN 104 / 113, which may be utilizing a NR radio technology, the CN 106 / 115 may also be in communication with another RAN (not shown) employing a GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0040] The CN 106 / 115 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or the other networks 112. The PSTN 108 may include circuit-switched telephone networks that provide plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the transmission control protocol (TCP), user datagram protocol (UDP) and / or the internet protocol (IP) in the TCP / IP internet protocol suite. The networks 112 may include wired and / or wireless communications networks owned and / or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs, which may employ the same RAT as the RAN 104 / 113 or a different RAT.

[0041] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links). For example, the WTRU 102c shown in FIG. 1 A may be configured to communicate with the base station 114a, which may employ a cellularbased radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[0042] FIG. 1 B is a system diagram illustrating an example WTRU 102. As shown in FIG. 1 B, the WTRU 102 may include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138, among others. It will be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0043] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. While FIG. 1 B depicts the processor 118 and the transceiver 120 as separate components, it will be appreciated that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0044] The transmit / receive element 122 may be configured to transmit signals to, or receive signals from, a base station (e.g., the base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In an embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF and light signals. It will be appreciated that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0045] Although the transmit / receive element 122 is depicted in FIG. 1 B as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0046] The transceiver 120 may be configured to modulate the signals that are to be transmitted by the transmit / receive element 122 and to demodulate the signals that are received by the transmit / receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11, for example.

[0047] The processor 118 of the WTRU 102 may be coupled to, and may receive user input data from, the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from, and store data in, any type of suitable memory, such as the non-removablememory 130 and / or the removable memory 132. The non-removable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from, and store data in, memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[0048] The processor 118 may receive power from the power source 134, and may be configured to distribute and / or control the power to the other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li- ion), etc.), solar cells, fuel cells, and the like.

[0049] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or in lieu of, the information from the GPS chipset 136, the WTRU 102 may receive location information over the air interface 116 from a base station (e.g., base stations 114a, 114b) and / or determine its location based on the timing of the signals being received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment.

[0050] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photographs and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a Virtual Reality and / or Augmented Reality (VR / AR) device, an activity tracker, and the like. The peripherals 138 may include one or more sensors, the sensors may be one or more of a gyroscope, an accelerometer, a hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0051] The WTRU 102 may include a full duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for both the UL (e.g., for transmission) and downlink(e.g . , for reception) may be concurrent and / or simultaneous. The full duplex radio may include an interference management unit to reduce and or substantially eliminate self-interference via either hardware (e.g., a choke) or signal processing via a processor (e.g., a separate processor (not shown) or via processor 118). In an embodiment, the WRTU 102 may include a half-duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for either the UL (e.g., for transmission) or the downlink (e.g., for reception)).

[0052] FIG. 1 C is a system diagram illustrating the RAN 104 and the CN 106 according to an embodiment. As noted above, the RAN 104 may employ an E-UTRA radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 104 may also be in communication with the CN 106.

[0053] The RAN 104 may include eNode-Bs 160a, 160b, 160c, though it will be appreciated that the RAN 104 may include any number of eNode-Bs while remaining consistent with an embodiment. The eNode-Bs 160a, 160b, 160c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the eNode-Bs 160a, 160b, 160c may implement MIMO technology. Thus, the eNode-B 160a, for example, may use multiple antennas to transmit wireless signals to, and / or receive wireless signals from, the WTRU 102a.

[0054] Each of the eNode-Bs 160a, 160b, 160c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the UL and / or DL, and the like. As shown in FIG. 1 C, the eNode-Bs 160a, 160b, 160c may communicate with one another over an X2 interface.

[0055] The CN 106 shown in FIG. 1C may include a mobility management entity (MME) 162, a serving gateway (SGW) 164, and a packet data network (PDN) gateway (or PGW) 166. While each of the foregoing elements are depicted as part of the CN 106, it will be appreciated that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0056] The MME 162 may be connected to each of the eNode-Bs 162a, 162b, 162c in the RAN 104 via an S1 interface and may serve as a control node. For example, the MME 162 may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, bearer activation / deactivation, selecting a particular serving gateway during an initial attach of the WTRUs 102a, 102b, 102c, and the like. The MME 162 may provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as GSM and / or WCDMA.

[0057] The SGW 164 may be connected to each of the eNode Bs 160a, 160b, 160c in the RAN 104 via the S1 interface. The SGW 164 may generally route and forward user data packets to / from the WTRUs 102a,102b, 102c. The SGW 164 may perform other functions, such as anchoring user planes during inter-eNode B handovers, triggering paging when DL data is available for the WTRUs 102a, 102b, 102c, managing and storing contexts of the WTRUs 102a, 102b, 102c, and the like.

[0058] The SGW 164 may be connected to the PGW 166, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices.

[0059] The CN 106 may facilitate communications with other networks. For example, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices. For example, the CN 106 may include, or may communicate with, an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that serves as an interface between the CN 106 and the PSTN 108. In addition, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to the other networks 112, which may include other wired and / or wireless networks that are owned and / or operated by other service providers.

[0060] Although the WTRU is described in FIGS. 1 A-1 D as a wireless terminal, it is contemplated that in certain representative embodiments that such a terminal may use (e.g., temporarily or permanently) wired communication interfaces with the communication network.

[0061] In representative embodiments, the other network 112 may be a WLAN.

[0062] A WLAN in Infrastructure Basic Service Set (BSS) mode may have an Access Point (AP) for the BSS and one or more stations (STAs) associated with the AP. The AP may have an access or an interface to a Distribution System (DS) or another type of wired / wireless network that carries traffic in to and / or out of the BSS. Traffic to STAs that originates from outside the BSS may arrive through the AP and may be delivered to the STAs. Traffic originating from STAs to destinations outside the BSS may be sent to the AP to be delivered to respective destinations. Traffic between STAs within the BSS may be sent through the AP, for example, where the source STA may send traffic to the AP and the AP may deliver the traffic to the destination STA. The traffic between STAs within a BSS may be considered and / or referred to as peer-to-peer traffic. The peer-to- peer traffic may be sent between (e.g., directly between) the source and destination STAs with a direct link setup (DLS). In certain representative embodiments, the DLS may use an 802.11e DLS or an 802.11z tunneled DLS (TDLS). A WLAN using an Independent BSS (IBSS) mode may not have an AP, and the STAs (e.g., all of the STAs) within or using the IBSS may communicate directly with each other. The IBSS mode of communication may sometimes be referred to herein as an “ad-hoc” mode of communication.

[0063] When using the 802.11ac infrastructure mode of operation or a similar mode of operations, the AP may transmit a beacon on a fixed channel, such as a primary channel. The primary channel may be a fixed width (e.g., 20 MHz wide bandwidth) or a dynamically set width via signaling. The primary channel may be the operating channel of the BSS and may be used by the STAs to establish a connection with the AP. In certain representative embodiments, Carrier Sense Multiple Access with Collision Avoidance (CSMA / CA) may be implemented, for example in in 802.11 systems. For CSMA / CA, the STAs (e.g., every STA), including the AP, may sense the primary channel. If the primary channel is sensed / detected and / or determined to be busy by a particular STA, the particular STA may back off. One STA (e.g., only one station) may transmit at any given time in a given BSS.

[0064] High Throughput (HT) STAs may use a 40 MHz wide channel for communication, for example, via a combination of the primary 20 MHz channel with an adjacent or nonadjacent 20 MHz channel to form a 40 MHz wide channel.

[0065] Very High Throughput (VHT) STAs may support 20MHz, 40 MHz, 80 MHz, and / or 160 MHz wide channels. The 40 MHz, and / or 80 MHz, channels may be formed by combining contiguous 20 MHz channels. A 160 MHz channel may be formed by combining 8 contiguous 20 MHz channels, or by combining two noncontiguous 80 MHz channels, which may be referred to as an 80-^0 configuration. For the 80+80 configuration, the data, after channel encoding, may be passed through a segment parser that may divide the data into two streams. Inverse Fast Fourier Transform (IFFT) processing, and time domain processing, may be done on each stream separately. The streams may be mapped on to the two 80 MHz channels, and the data may be transmitted by a transmitting STA. At the receiver of the receiving STA, the above described operation for the 80+80 configuration may be reversed, and the combined data may be sent to the Medium Access Control (MAC).

[0066] Sub 1 GHz modes of operation are supported by 802.11 af and 802.11 ah. The channel operating bandwidths, and carriers, are reduced in 802.11 af and 802.11 ah relative to those used in 802.11 n, and 802.11 ac. 802.11 af supports 5 MHz, 10 MHz and 20 MHz bandwidths in the TV White Space (TVWS) spectrum, and 802.11 ah supports 1 MHz, 2 MHz, 4 MHz, 8 MHz, and 16 MHz bandwidths using non-TVWS spectrum. According to a representative embodiment, 802.11 ah may support Meter Type Control / Machine- Type Communications, such as MTC devices in a macro coverage area. MTC devices may have certain capabilities, for example, limited capabilities including support for (e.g., only support for) certain and / or limited bandwidths. The MTC devices may include a battery with a battery life above a threshold (e.g., to maintain a very long battery life).

[0067] WLAN systems, which may support multiple channels, and channel bandwidths, such as 802.11 n, 802.11ac, 802.11 at, and 802.11 ah, include a channel which may be designated as the primary channel. The primary channel may have a bandwidth equal to the largest common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel may be set and / or limited by a STA, from among all STAs in operating in a BSS, which supports the smallest bandwidth operating mode. In the example of 802.11 ah, the primary channel may be 1 MHz wide for STAs (e.g., MTC type devices) that support (e.g., only support) a 1 MHz mode, even if the AP, and other STAs in the BSS support 2 MHz, 4 MHz, 8 MHz, 16 MHz, and / or other channel bandwidth operating modes. Carrier sensing and / or Network Allocation Vector (NAV) settings may depend on the status of the primary channel. If the primary channel is busy, for example, due to a STA (which supports only a 1 MHz operating mode), transmitting to the AP, the entire available frequency bands may be considered busy even though a majority of the frequency bands remains idle and may be available.

[0068] In the United States, the available frequency bands, which may be used by 802.11 ah, are from 902 MHz to 928 MHz. In Korea, the available frequency bands are from 917.5 MHz to 923.5 MHz. In Japan, the available frequency bands are from 916.5 MHz to 927.5 MHz. The total bandwidth available for 802.11 ah is 6 MHz to 26 MHz depending on the country code.

[0069] FIG. 1 D is a system diagram illustrating the RAN 113 and the CN 115 according to an embodiment. As noted above, the RAN 113 may employ an NR radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 113 may also be in communication with the CN 115.

[0070] The RAN 113 may include gNBs 180a, 180b, 180c, though it will be appreciated that the RAN 113 may include any number of gNBs while remaining consistent with an embodiment. The gNBs 180a, 180b, 180c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the gNBs 180a, 180b, 180c may implement MIMO technology. For example, gNBs 180a, 108b may utilize beamforming to transmit signals to and / or receive signals from the gNBs 180a, 180b, 180c. Thus, the gNB 180a, for example, may use multiple antennas to transmit wireless signals to, and / or receive wireless signals from, the WTRU 102a. In an embodiment, the gNBs 180a, 180b, 180c may implement carrier aggregation technology. For example, the gNB 180a may transmit multiple component carriers to the WTRU 102a (not shown). A subset of these component carriers may be on unlicensed spectrum while the remaining component carriers may be on licensed spectrum. In an embodiment, the gNBs 180a, 180b, 180c may implement Coordinated Multi-Point (CoMP) technology. For example, WTRU 102a may receive coordinated transmissions from gNB 180a and gNB 180b (and / or gNB 180c).

[0071] The WTRUs 102a, 102b, 102c may communicate with gNBs 180a, 180b, 180c using transmissions associated with a scalable numerology. For example, the OFDM symbol spacing and / or OFDM subcarrier spacing may vary for different transmissions, different cells, and / or different portions of the wireless transmission spectrum. The WTRUs 102a, 102b, 102c may communicate with gNBs 180a, 180b, 180c using subframe or transmission time intervals (TTIs) of various or scalable lengths (e.g., containing varying number of OFDM symbols and / or lasting varying lengths of absolute time).

[0072] The gNBs 180a, 180b, 180c may be configured to communicate with the WTRUs 102a, 102b, 102c in a standalone configuration and / or a non-standalone configuration. In the standalone configuration, WTRUs 102a, 102b, 102c may communicate with gNBs 180a, 180b, 180c without also accessing other RANs (e.g., such as eNode-Bs 160a, 160b, 160c). In the standalone configuration, WTRUs 102a, 102b, 102c may utilize one or more of gNBs 180a, 180b, 180c as a mobility anchor point. In the standalone configuration, WTRUs 102a, 102b, 102c may communicate with gNBs 180a, 180b, 180c using signals in an unlicensed band. In a non-standalone configuration WTRUs 102a, 102b, 102c may communicate with / connect to gNBs 180a, 180b, 180c while also communicating with / connecting to another RAN such as eNode-Bs 160a, 160b, 160c. For example, WTRUs 102a, 102b, 102c may implement DC principles to communicate with one or more gNBs 180a, 180b, 180c and one or more eNode-Bs 160a, 160b, 160c substantially simultaneously. In the non- standalone configuration, eNode-Bs 160a, 160b, 160c may serve as a mobility anchor for WTRUs 102a, 102b, 102c and gNBs 180a, 180b, 180c may provide additional coverage and / or throughput for servicing WTRUs 102a, 102b, 102c.

[0073] Each of the gNBs 180a, 180b, 180c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the UL and / or DL, support of network slicing, dual connectivity, interworking between NR and E-UTRA, routing of user plane data towards User Plane Function (UPF) 184a, 184b, routing of control plane information towards Access and Mobility Management Function (AMF) 182a, 182b and the like. As shown in FIG. 1 D, the gNBs 180a, 180b, 180c may communicate with one another over an Xn interface.

[0074] The CN 115 shown in FIG. 1 D may include at least one AMF 182a, 182b, at least one UPF 184a, 184b, at least one Session Management Function (SMF) 183a, 183b, and possibly a Data Network (DN) 185a, 185b. While each of the foregoing elements are depicted as part of the CN 115, it will be appreciated that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0075] The AMF 182a, 182b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via an N2 interface and may serve as a control node. For example, the AMF 182a, 182b may beresponsible for authenticating users of the WTRUs 102a, 102b, 102c, support for network slicing (e.g., handling of different PDU sessions with different requirements), selecting a particular SMF 183a, 183b, management of the registration area, termination of NAS signaling, mobility management, and the like. Network slicing may be used by the AMF 182a, 182b in order to customize CN support for WTRUs 102a, 102b, 102c based on the types of services being utilized WTRUs 102a, 102b, 102c. For example, different network slices may be established for different use cases such as services relying on ultra-reliable low latency (URLLC) access, services relying on enhanced massive mobile broadband (eMBB) access, services for machine type communication (MTC) access, and / or the like. The AMF 162 may provide a control plane function for switching between the RAN 113 and other RANs (not shown) that employ other radio technologies, such as LTE, LTE-A, LTE-A Pro, and / or non-3GPP access technologies such as WiFi.

[0076] The SMF 183a, 183b may be connected to an AMF 182a, 182b in the CN 115 via an N11 interface. The SMF 183a, 183b may also be connected to a UPF 184a, 184b in the CN 115 via an N4 interface. The SMF 183a, 183b may select and control the UPF 184a, 184b and configure the routing of traffic through the UPF 184a, 184b. The SMF 183a, 183b may perform other functions, such as managing and allocating UE IP address, managing PDU sessions, controlling policy enforcement and QoS, providing downlink data notifications, and the like. A PDU session type may be IP-based, non-IP based, Ethernet-based, and the like.

[0077] The UPF 184a, 184b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via an N3 interface, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP- enabled devices. The UPF 184, 184b may perform other functions, such as routing and forwarding packets, enforcing user plane policies, supporting multi-homed PDU sessions, handling user plane QoS, buffering downlink packets, providing mobility anchoring, and the like.

[0078] The CN 115 may facilitate communications with other networks. For example, the CN 115 may include, or may communicate with, an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that serves as an interface between the CN 115 and the PSTN 108. In addition, the CN 115 may provide the WTRUs 102a, 102b, 102c with access to the other networks 112, which may include other wired and / or wireless networks that are owned and / or operated by other service providers. In one embodiment, the WTRUs 102a, 102b, 102c may be connected to a local Data Network (DN) 185a, 185b through the UPF 184a, 184b via the N3 interface to the UPF 184a, 184b and an N6 interface between the UPF 184a, 184b and the DN 185a, 185b.

[0079] In view of Figures 1 A-1 D, and the corresponding description of Figures 1 A-1 D, one or more, or all, of the functions described herein with regard to one or more of: WTRU 102a-d, Base Station 114a-b, eNode-B160a-c, MME 162, SGW 164, PGW 166, gNB 180a-c, AMF 182a-b, UPF 184a-b, SMF 183a-b, DN 185a-b, and / or any other device(s) described herein, may be performed by one or more emulation devices (not shown). The emulation devices may be one or more devices configured to emulate one or more, or all, of the functions described herein. For example, the emulation devices may be used to test other devices and / or to simulate network and / or WTRU functions.

[0080] The emulation devices may be designed to implement one or more tests of other devices in a lab environment and / or in an operator network environment. For example, the one or more emulation devices may perform the one or more, or all, functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network in order to test other devices within the communication network. The one or more emulation devices may perform the one or more, or all, functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. The emulation device may be directly coupled to another device for purposes of testing and / or may performing testing using over-the-air wireless communications.

[0081] The one or more emulation devices may perform the one or more, including all, functions while not being implemented / deployed as part of a wired and / or wireless communication network. For example, the emulation devices may be utilized in a testing scenario in a testing laboratory and / or a non-deployed (e.g., testing) wired and / or wireless communication network in order to implement testing of one or more components. The one or more emulation devices may be test equipment. Direct RF coupling and / or wireless communications via RF circuitry (e.g., which may include one or more antennas) may be used by the emulation devices to transmit and / or receive data.

[0082] This application describes a variety of aspects, including tools, features, examples, models, approaches, etc. Many of these aspects are described with specificity and, at least to show the individual characteristics, are often described in a manner that may sound limiting. However, this is for purposes of clarity in description, and does not limit the application or scope of those aspects. Indeed, all of the different aspects may be combined and interchanged to provide further aspects. Moreover, the aspects may be combined and interchanged with aspects described in earlier filings as well.

[0083] The aspects described and contemplated in this application may be implemented in many different forms. FIGS. 5-10 described herein may provide some examples, but other examples are contemplated. The discussion of FIGS. 5-10 does not limit the breadth of the implementations. At least one of the aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a bitstream generated or encoded. These and other aspects may be implemented as a method,an apparatus, a computer readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the methods described, and / or a computer readable storage medium having stored thereon a bitstream generated according to any of the methods described.

[0084] In the present application, the terms “reconstructed” and “decoded” may be used interchangeably, the terms “pixel” and “sample” may be used interchangeably, the terms “image,” “picture” and “frame” may be used interchangeably.

[0085] Various methods are described herein, and each of the methods comprises one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as “first”, “second”, etc. may be used in various examples to modify an element, component, step, operation, etc., such as, for example, a “first decoding” and a “second decoding”. Use of such terms does not imply an ordering to the modified operations unless specifically required. So, in this example, the first decoding need not be performed before the second decoding, and may occur, for example, before, during, or in an overlapping time period with the second decoding.

[0086] Various methods and other aspects described in this application may be used to modify modules, for example, decoding modules, of a video encoder 200 and decoder 300 as shown in FIG. 2 and FIG. 3. Moreover, the subject matter disclosed herein may be applied, for example, to any type, format or version of video coding, whether described in a standard or a recommendation, whether pre-existing or future-developed, and extensions of any such standards and recommendations. Unless indicated otherwise, or technically precluded, the aspects described in this application may be used individually or in combination.

[0087] Various numeric values are used in examples described the present application. These and other specific values are for purposes of describing examples and the aspects described are not limited to these specific values.

[0088] FIG. 2 is a diagram showing an example video encoder. Variations of example encoder 200 are contemplated, but the encoder 200 is described below for purposes of clarity without describing all expected variations.

[0089] Before being encoded, the video sequence may go through pre-encoding processing (201), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to get a signal distribution more resilient to compression (for instance using a histogram equalization of one of the color components). Metadata may be associated with the pre-processing, and attached to the bitstream.

[0090] In the encoder 200, a picture is encoded by the encoder elements as described below. The picture to be encoded is partitioned (202) and processed in units of, for example, coding units (CUs). Each unit is encoded using, for example, either an intra or inter mode. When a unit is encoded in an intra mode, it performs intra prediction (260). In an inter mode, motion estimation (275) and compensation (270) are performed. The encoder decides (205) which one of the intra mode or inter mode to use for encoding the unit, and indicates the intra / inter decision by, for example, a prediction mode flag. Prediction residuals are calculated, for example, by subtracting (210) the predicted block from the original image block.

[0091] The prediction residuals are then transformed (225) and quantized (230). The quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy coded (245) to output a bitstream. The encoder can skip the transform and apply quantization directly to the non-transformed residual signal. The encoder can bypass both transform and quantization, i.e., the residual is coded directly without the application of the transform or quantization processes.

[0092] The encoder decodes an encoded block to provide a reference for further predictions. The quantized transform coefficients are de-quantized (240) and inverse transformed (250) to decode prediction residuals. Combining (255) the decoded prediction residuals and the predicted block, an image block is reconstructed. In-loop filters (265) are applied to the reconstructed picture to perform, for example, deblocking / SAO (Sample Adaptive Offset) filtering to reduce encoding artifacts. The filtered image is stored at a reference picture buffer (280).

[0093] FIG. 3 is a diagram showing an example of a video decoder. In example decoder 300, a bitstream is decoded by the decoder elements as described below. Video decoder 300 generally performs a decoding pass reciprocal to the encoding pass as described in FIG. 2. The encoder 200 also generally performs video decoding as part of encoding video data.

[0094] In particular, the input of the decoder includes a video bitstream, which may be generated by video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, motion vectors, and other coded information. The picture partition information indicates how the picture is partitioned. The decoder may therefore divide (335) the picture according to the decoded picture partitioning information. The transform coefficients are de-quantized (340) and inverse transformed (350) to decode the prediction residuals.Combining (355) the decoded prediction residuals and the predicted block, an image block is reconstructed. The predicted block may be obtained (370) from intra prediction (360) or motion-compensated prediction (i.e., inter prediction) (375). In-loop filters (365) are applied to the reconstructed image. The filtered image is stored at a reference picture buffer (380).

[0095] The decoded picture can further go through post-decoding processing (385), for example, an inverse color transform (e.g. conversion from YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping performing the inverse of the remapping process performed in the pre-encoding processing (201). The post-decoding processing can use metadata derived in the pre-encoding processing and signaled in the bitstream. In an example, the decoded images (e.g., after application of the in-loop filters (365) and / or after post-decoding processing (385), if post-decoding processing is used) may be sent to a display device for rendering to a user.

[0096] FIG. 4 is a diagram showing an example of a system in which various aspects and examples described herein may be implemented. System 400 may be embodied as a device including the various components described below and is configured to perform one or more of the aspects described in this document. Examples of such devices, include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 400, singly or in combination, may be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one example, the processing and encoder / decoder elements of system 400 are distributed across multiple ICs and / or discrete components. In various examples, the system 400 is communicatively coupled to one or more other systems, or other electronic devices, via, for example, a communications bus or through dedicated input and / or output ports. In various examples, the system 400 is configured to implement one or more of the aspects described in this document.

[0097] The system 400 includes at least one processor 410 configured to execute instructions loaded therein for implementing, for example, the various aspects described in this document. Processor 410 can include embedded memory, input output interface, and various other circuitries as known in the art. The system 400 includes at least one memory 420 (e.g., a volatile memory device, and / or a non-volatile memory device). System 400 includes a storage device 440, which can include non-volatile memory and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drive, and / or optical disk drive. The storage device 440 can include an internal storage device, an attached storage device (including detachable and non-detachable storage devices), and / or a network accessible storage device, as non-limiting examples.

[0098] System 400 includes an encoder / decoder module 430 configured, for example, to process data to provide an encoded video or decoded video, and the encoder / decoder module 430 can include its ownprocessor and memory. The encoder / decoder module 430 represents module(s) that may be included in a device to perform the encoding and / or decoding functions. As is known, a device can include one or both of the encoding and decoding modules. Additionally, encoder / decoder module 430 may be implemented as a separate element of system 400 or may be incorporated within processor 410 as a combination of hardware and software as known to those skilled in the art.

[0099] Program code to be loaded onto processor 410 or encoder / decoder 430 to perform the various aspects described in this document may be stored in storage device 440 and subsequently loaded onto memory 420 for execution by processor 410. In accordance with various examples, one or more of processor 410, memory 420, storage device 440, and encoder / decoder module 430 can store one or more of various items during the performance of the processes described in this document. Such stored items can include, but are not limited to, the input video, the decoded video or portions of the decoded video, the bitstream, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0100] In some examples, memory inside of the processor 410 and / or the encoder / decoder module 430 is used to store instructions and to provide working memory for processing that is needed during encoding or decoding. In other examples, however, a memory external to the processing device (for example, the processing device may be either the processor 410 or the encoder / decoder module 430) is used for one or more of these functions. The external memory may be the memory 420 and / or the storage device 440, for example, a dynamic volatile memory and / or a non-volatile flash memory. In several examples, an external non-volatile flash memory is used to store the operating system of, for example, a television. In at least one example, a fast external dynamic volatile memory such as a RAM is used as working memory for video encoding and decoding operations.

[0101] The input to the elements of system 400 may be provided through various input devices as indicated in block 445. Such input devices include, but are not limited to, (i) a radio frequency (RF) portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Component (COMP) input terminal (or a set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Other examples, not shown in FIG. 4, include composite video.

[0102] In various examples, the input devices of block 445 have associated respective input processing elements as known in the art. For example, the RF portion may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) downconverting the selected signal, (iii) band-limiting again to a narrower band of frequencies coto select (for example) a signal frequency band which may be referred to as a channel in certain examples, (iv) demodulating the downconverted and band-limited signal, (v) performing error correction, and / or (vi) demultiplexing to select the desired stream of data packets. The RF portion of various examples includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion can include a tuner that performs various of these functions, including, for example, downconverting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box example, the RF portion and its associated input processing element receives an RF signal transmitted over a wired (for example, cable) medium, and performs frequency selection by filtering, downconverting, and filtering again to a desired frequency band. Various examples rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements performing similar or different functions. Adding elements can include inserting elements in between existing elements, such as, for example, inserting amplifiers and an analog-to-digital converter. In various examples, the RF portion includes an antenna.

[0103] The USB and / or HDMI terminals can include respective interface processors for connecting system 400 to other electronic devices across USB and / or HDMI connections. It is to be understood that various aspects of input processing, for example, Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or within processor 410 as necessary. Similarly, aspects of USB or HDMI interface processing may be implemented within separate interface ICs or within processor 410 as necessary. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 410, and encoder / decoder 430 operating in combination with the memory and storage elements to process the datastream as necessary for presentation on an output device.

[0104] Various elements of system 400 may be provided within an integrated housing, Within the integrated housing, the various elements may be interconnected and transmit data therebetween using suitable connection arrangement 425, for example, an internal bus as known in the art, including the Inter-IC (I2C) bus, wiring, and printed circuit boards.

[0105] The system 400 includes communication interface 450 that enables communication with other devices via communication channel 460. The communication interface 450 can include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 460. The communication interface 450 can include, but is not limited to, a modem or network card and the communication channel 460 may be implemented, for example, within a wired and / or a wireless medium.

[0106] Data is streamed, or otherwise provided, to the system 400, in various examples, using a wireless network such as a Wi-Fi network, for example IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal of these examples is received over the communications channel 460 and the communications interface 450 which are adapted for Wi-Fi communications. The communications channel 460 of these examples is typically connected to an access point or router that provides access to external networks including the Internet for allowing streaming applications and other over-the-top communications. Other examples provide streamed data to the system 400 using a set-top box that delivers the data over the HDMI connection of the input block 445. Still other examples provide streamed data to the system 400 using the RF connection of the input block 445. As indicated above, various examples provide data in a non-streaming manner. Additionally, various examples use wireless networks other than Wi-Fi, for example a cellular network or a Bluetooth® network.

[0107] The system 400 can provide an output signal to various output devices, including a display 475, speakers 485, and other peripheral devices 495. The display 475 of various examples includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 475 may be for a television, a tablet, a laptop, a cell phone (mobile phone), or other device. The display 475 can also be integrated with other components (for example, as in a smart phone), or separate (for example, an external monitor for a laptop). The other peripheral devices 495 include, in various examples, one or more of a stand-alone digital video disc (or digital versatile disc) (DVD, for both terms), a disk player, a stereo system, and / or a lighting system. Various examples use one or more peripheral devices 495 that provide a function based on the output of the system 400. For example, a disk player performs the function of playing the output of the system 400.

[0108] In various examples, control signals are communicated between the system 400 and the display 475, speakers 485, or other peripheral devices 495 using signaling such as AV. Link, Consumer Electronics Control (CEC), or other communications protocols that enable device-to-device control with or without user intervention. The output devices may be communicatively coupled to system 400 via dedicated connections through respective interfaces 470, 480, and 490. Alternatively, the output devices may be connected to system 400 using the communications channel 460 via the communications interface 450. The display 475 and speakers 485 may be integrated in a single unit with the other components of system 400 in an electronic device such as, for example, a television. In various examples, the display interface 470 includes a display driver, such as, for example, a timing controller (T Con) chip.

[0109] The display 475 and speakers 485 can alternatively be separate from one or more of the other components, for example, if the RF portion of input 445 is part of a separate set-top box. In various examples in which the display 475 and speakers 485 are external components, the output signal may be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0110] The examples may be carried out by computer software implemented by the processor 410 or by hardware, or by a combination of hardware and software. As a non-limiting example, the examples may be implemented by one or more integrated circuits. The memory 420 may be of any type appropriate to the technical environment and may be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. The processor 410 may be of any type appropriate to the technical environment, and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as non-limiting examples.

[0111] Various implementations involve decoding. “Decoding”, as used in this application, can encompass all or part of the processes performed, for example, on a received encoded sequence in order to produce a final output suitable for display. In various examples, such processes include one or more of the processes typically performed by a decoder, for example, entropy decoding, inverse quantization, inverse transformation, and differential decoding. In various examples, such processes also, or alternatively, include processes performed by a decoder of various implementations described in this application.

[0112] As further examples, in one example “decoding” refers only to entropy decoding, in another example “decoding” refers only to differential decoding, and in another example “decoding” refers to a combination of entropy decoding and differential decoding. Whether the phrase “decoding process” is intended to refer specifically to a subset of operations or generally to the broader decoding process will be clear based on the context of the specific descriptions and is believed to be well understood by those skilled in the art.

[0113] Various implementations involve encoding. In an analogous way to the above discussion about “decoding”, “encoding” as used in this application can encompass all or part of the processes performed, for example, on an input video sequence in order to produce an encoded bitstream. In various examples, such processes include one or more of the processes typically performed by an encoder, for example, partitioning, differential encoding, transformation, quantization, and entropy encoding. In various examples, such processes also, or alternatively, include processes performed by an encoder of various implementations described in this application.

[0114] As further examples, in one example “encoding” refers only to entropy encoding, in another example “encoding” refers only to differential encoding, and in another example “encoding” refers to a combination of differential encoding and entropy encoding. Whether the phrase “encoding process” is intended to refer specifically to a subset of operations or generally to the broader encoding process will be clear based on the context of the specific descriptions and is believed to be well understood by those skilled in the art.

[0115] Note that syntax elements as used herein are descriptive terms. As such, they do not preclude the use of other syntax element names.

[0116] When a figure is presented as a flow diagram, it should be understood that it also provides a block diagram of a corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow diagram of a corresponding method / process.

[0117] The implementations and aspects described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed can also be implemented in other forms (for example, an apparatus or program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end-users.

[0118] Reference to “one example” or “an example” or “one implementation” or “an implementation”, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the example is included in at least one example. Thus, the appearances of the phrase “in one example” or “in an example” or “in one implementation” or “in an implementation”, as well any other variations, appearing in various places throughout this application are not necessarily all referring to the same example.

[0119] Additionally, this application may refer to “determining” various pieces of information. Determining the information can include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory. Obtaining may include receiving, retrieving, constructing, generating, and / or determining.

[0120] Further, this application may refer to “accessing” various pieces of information. Accessing the information can include one or more of, for example, receiving the information, retrieving the information (forexample, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0121] Additionally, this application may refer to “receiving” various pieces of information. Receiving is, as with “accessing”, intended to be a broad term. Receiving the information can include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further, “receiving” is typically involved, in one way or another, during operations such as, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0122] It is to be appreciated that the use of any of the following ”, “and / or”, and “at least one of”, for example, in the cases of “A / B”, “A and / or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended, as is clear to one of ordinary skill in this and related arts, for as many items as are listed.

[0123] Also, as used herein, the word “signal” refers to, among other things, indicating something to a corresponding decoder. In this way, in an example the same parameter is used at both the encoder side and the decoder side. Thus, for example, an encoder can transmit (explicit signaling) a particular parameter to the decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter as well as others, then signaling may be used without transmitting (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding transmission of any actual functions, a bit savings is realized in various examples. It is to be appreciated that signaling may be accomplished in a variety of ways. For example, one or more syntax elements, flags, and so forth are used to signal information to a corresponding decoder in various examples. While the preceding relates to the verb form of the word “signal”, the word “signal” can also be used herein as a noun.

[0124] As will be evident to one of ordinary skill in the art, implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted. The information can include, forexample, instructions for performing a method, or data produced by one of the described implementations. For example, a signal may be formatted to carry the bitstream of a described example. Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on, or accessed or received from, a processor-readable medium.

[0125] Many examples are described herein. Features of examples may be provided alone or in any combination, across various claim categories and types. Further, examples may include one or more of the features, devices, or aspects described herein, alone or in any combination, across various claim categories and types. For example, features described herein may be implemented in a bitstream or signal that includes information generated as described herein. The information may allow a decoder to decode a bitstream, the encoder, bitstream, and / or decoder according to any of the embodiments described. For example, features described herein may be implemented by creating and / or transmitting and / or receiving and / or decoding a bitstream or signal. For example, features described herein may be implemented a method, process, apparatus, medium storing instructions, medium storing data, or signal. For example, features described herein may be implemented by a TV, set-top box, cell phone, tablet, or other electronic device that performs decoding. The TV, set-top box, cell phone, tablet, or other electronic device may display (e.g. using a monitor, screen, or other type of display) a resulting image (e.g., an image from residual reconstruction of the video bitstream). The TV, set-top box, cell phone, tablet, or other electronic device may receive a signal including an encoded image and perform decoding.

[0126] Haptics may provide multimedia application users (e.g., referred to herein as users or end users) with sensory effects in addition to audio and video allowing the multimedia application users to experience, for example, sensory immersion. Users may enjoy media more abundantly through haptics effects. Some media container formats, such as ISO based media file format (ISOBMFF), may define haptics as one of the supported primary media types (e.g., along with the other primary media types, such as audio and / or video). The data format, metadata, and / or codec architecture for haptics signals may be defined. Haptics data may be transmitted to devices that act as actuators and may provide information for actuators to operate (e.g., according to the values contained in the haptics data).

[0127] MPEG-I Haptic Streaming (MIHS) may be a streaming format that may be used to transport haptics data. Haptics data may include haptic effects. Haptics data may be packetized, for example, based on theMIHS format and may be delivered to actuators that operate according to the haptics effects. MIHS may have two levels of packetization (e.g., MIHS units and MIHS packets).

[0128] MIHS units may include a MIHS unit header and zero or more MIHS packets. MIHS units may belong to one of four types (e.g., initialization MIHS, temporal MIHS, spatial MIHS, and silent MIHS). The initialization MIHS unit may set a timestamp and may include one or more MIHS packets. A temporal MIHS unit may contain data defining time dependent effects and may support modalities (e.g., pressure, velocity, and / or acceleration. A spatial MIHS unit may contain time independent haptic effects and support modalities (e.g., vibrotactile texture, stiffness, and / or friction). Temporal and spatial MIHS units may include one or more MIHS packets. Silent MIHS units may be transmitted, for example, when there is no haptics data to transmit and no MIHS packets.

[0129] A temporal MIHS unit may be a sync unit or a non-sync unit. A non-sync MIHS unit may depend on the most recent sync MIHS unit for decoding. Initialization MIHS units may be transmitted regularly to allow for random access (e.g., because initialization MIHS units may be essential to decode the haptics stream).

[0130] A MIHS format may be a self-contained stream format for transporting MPEG-I haptic data.

[0131] A MIHS packet may be a unit of haptics data which includes metadata or binary effect data.

[0132] A MIHS unit may be a unit of haptics data which includes metadata and / or haptic effect data. The haptics data format may be divided into MIHS Unit and Packet, and MIHS unit may have 4 types.

[0133] An initialization unit may be a type of MIHS unit containing metadata necessary to reset and initialize a haptic decoder.

[0134] A temporal unit may contain one or more MIHS packets. The duration of a temporal unit may be a positive number.

[0135] A spatial unit may contain one or more MIHS packets. The duration of a spatial unit may be zero.

[0136] A silent unit may indicate that there may be no haptic effect that starts during this time interval and may not include any MIHS packets. The duration of a silent unit may be a positive number.

[0137] A MIHS Layer may be a numerical parameter associated with a MIHS packet, for example, that may be used to represent the importance of the packet for scalability.

[0138] Real-time transport protocol (RTP) may be a standard protocol for transmitting real-time voice, video, and / or data over an IP network. RTP may use real-time control protocol (RTCP), for example to monitor data transfer status, provide minimum control functions, and / or provide media identification. RTP may be used in telecommunications and entertainment systems involving streaming media, such as telephony, WebRTC,television services, and / or video telephony including web-based push-to-talk capabilities. The value of utilization may increase when a separate payload is configured according to the video codec. Several RTP payload standards may be adapted to the characteristics of specific codecs.

[0139] Haptics may be a new top-level media type (e.g., at the same level as video and audio). As with the other media types, haptics may be streamed in real-time or near real-time (e.g., among other streaming alternatives), for example when used for live streaming or real-time communication services. Real-time streaming may use protocols such as RTP, SRTP, RSTP, MOQ, ROQ, and others. Live streaming capabilities may be improved (e.g., as described herein), for example using RTP and / or related protocols, such as SDP, as an exemplary base protocol. It should be appreciated that the mechanisms described herein may be applicable to other live streaming protocols as well.

[0140] Compared to other top-level media types (e.g., audio and video), haptics may have a more diverse set of rendering devices and / or types of data corresponding to these devices (e.g., since haptics may cover a large range of perceptions, including pressure, acceleration, velocity, vibration, temperature, force, friction, texture, etc.). Haptic data streams may be comparable in size to audio streams (e.g., within a few hundreds of kilobits per second). Some applications may use a range of haptic devices and perceptions, which may result in high network resource usages. It may be desirable to limit network resource usage and to reduce the impact of network congestion and loss on the quality of experience (QoE) of applications making use of haptics.

[0141] Haptics data may be communicated between data sources such as application servers and / or haptics sensors, towards devices capable of rendering haptics signals, such as mobile phones, gaming consoles, and / or XR devices. Haptics data may be combined with other media data such as video and audio (e.g., to enable a realistic experience).

[0142] Haptic services may require various types of receivers (e.g., devices) to provide a realistic experience to the user. The main role of a receiver may be to receive haptic data and use the haptic data to play various haptics effects. Each device may have different specifications, for example depending on usage. There may be parameters (e.g., 15 parameters) which may be related to the device. A parameter (e.g., each parameter) may have various option(s). For example, in the case of frequency, each device may have different requirements for a preferred frequency range (e.g., even if each device has the same or similar functions). As technology advances and new effects emerge, the requirement of haptic devices may increase (e.g., continuously). Communication methods (e.g., as described herein) may select what haptic data may be sent and / or to mark the haptic data sent (e.g., to enable the sender and the network to provide differentiated QoS on the flow to meet various requirements reflecting the haptics device characteristics).

[0143] Communication of haptic data may be over wireless radio access networks, which may in some cases be unstable or have limited bandwidth. A haptics data communication system may, therefore, experience network congestion due to the underlying network conditions, and / or due to sharing limited bandwidth with other real-time media flows. When congestion occurs, some of the haptics media packets may be dropped and the QoE for the user may be impacted. For example, a perceived vibration may be intermittent, texture perception may change to a different texture, perception of friction may be inconsistent, etc. This may result in a reduction of the realism of a game or application experience (e.g., for the user). The effect of congestion on haptics QoE for end users may be minimized (e.g., as described herein).

[0144] Packet loss may negatively impact an end-user’s haptics QoE. For example, when bandwidth is limited systematic retransmission of all lost packets may not be a viable option. A selective retransmission of the packets (e.g., only the packets that are most important for the haptics QoE of the end users) may be performed.

[0145] Feature(s) associated with efficient transmission of haptic data are provided herein.

[0146] The data transmission environment may include one or more of the following: a media sender; a media receiver; an intermediate node; or a device. Each entity may be a general internet device and / or may be a cellular network node. FIG. 5 illustrates an example of a system for efficiently transmitting haptic data.

[0147] A media sender and media receiver may perform a media session negotiation procedure, for example for identifying and selecting / marking the most important portion of the haptic traffic (e.g., using SDP to determine the identification / selection / marking method).

[0148] The media sender (e.g., an application server (AS) and / or user equipment (UE)) may transmit media flows, including a haptics media flow. A haptic media flow may be composed of haptics media packet(s).

[0149] An intermediate node (e.g., UPF, a UE, MUC, SFU, a MOQ or ROQ proxy, an RTP middle box) may be on a path of the haptics media flow. The intermediate node may access some metadata associated with the haptics media flow (e.g., haptics media transport lEs as described herein). For example, the intermediate node may access some metadata associated with the haptics media flow because the metadata may be transmitted in cleartext and / or because the intermediate node may be a trusted intermediate node on the media flow path.

[0150] A media receiver (e.g., UE) may receive media flows, including a haptics media flow.

[0151] Haptic transport information elements (lEs) may be used to support efficient transmission of haptic data. There may be multiple (e.g., two) types of haptic transport lEs. The lEs may be defined to increase theQoE of a haptics-based service. The lEs may be used according to the policy defined by network operators, application providers, and / or end users.

[0152] Media session negotiation may occur (e.g., between a sender and a receiver) using per-flow haptic transport lEs, for example to support haptic media packet transmission (e.g., as shown at 1 in FIG. 5).

[0153] Network congestion may be efficiently handled by determining the priority of data based on the per packet haptic transport lEs (e.g., as defined herein). Packet transmission efficiency may be increased by processing the packets stored in the queue according to a defined algorithm (e.g., as shown at 2 in FIG. 5). Packet transmission efficiency may minimize the effects of congestion on QoE.

[0154] When a packet is lost, a process of detecting loss and selectively requesting packet retransmission may be used, as defined herein. Packet retransmission may minimize the effects of packet loss on QoE (e.g., as shown at 3 in FIG. 5).

[0155] A haptics media sender (also referred to herein as a media sender or sender) may be a node that generates and transmits media flows to a receiving node which may be called a haptics media receiver or a haptics rendering device (also referred to herein as a media receiver or a receiver). A haptics media flow (e.g., an RTP or MOQ flow) may be composed of haptic media packets (e.g., RTP Packet, MOQ messages, and / or other media format over an IP packet and / or UDP datagram). The haptic media packet may be transmitted from haptic media sender to haptics media receiver through intermediate node(s). The haptics media receiver may depacketize the haptics data and may transmit it to devices (e.g., actuator(s)) which may decode and / or render the haptics effects.

[0156] A haptics transport unit (HT Unit or HTU) may be effectively a unit of information usable by the application. An HTU may be a unit of transport of information, which may be split during the transport (e.g., as fragments in different RTP packets) and may be reassembled for use by the receiving application. For example, an HTU for haptics applications may be a MIHS unit. For example, an HTU for haptics may be a MIHS packet. The terms HTU, HT unit, and MIHS unit are used interchangeably herein.

[0157] A haptics media packet may be a PDU that carries one or more HTUs or a fragment of an HTU. For example, a haptics media packet may be an RTP packet carrying an HTU, multiple HTUs, and / or an HTU fragment. The terms haptics media packet, haptics RTP packet, RTP packet are used interchangeably herein.

[0158] Intermediate nodes may be a node on the path of the haptics media flow. An intermediate node may be a UPF, a UE, MUC, SFU, a MOQ or ROQ proxy, or an RTP middle box. For example, an intermediate node may be a media receiver (e.g., UE), for example that may be tethered to a haptics device and may forwardhaptics media packets between the haptics device and a media sender in the network. The intermediate node may be an application endpoint (e.g., WTRU including a haptics device).

[0159] Feature(s) associated with haptics transport are provided herein.

[0160] HT units (e.g., MIHS units) may be units of payload for RTP packets. A new RTP payload header for haptics may describe (e.g., be needed to describe) the content of the RTP payload, and may enable nodes (e.g., intermediate, sender, receiver) to apply any needed network service (e.g., identifying / selecting / marking packets, congestion control, and / or loss recovery) while sending or forwarding RTP PDUs. An RTP payload header may enable a node to provide a network service without requiring the node to parse the payload. Parsing the payload may not be possible (e.g., decoder not available on the node or payload not accessible / encrypted or payload partially inaccessible / fragmented) or may be costly (e.g., in terms of processing resources on the node). HT units may be larger than the maximum that fit in an RTP PDU. In examples, an HT Unit may be fragmented across multiple RTP PDUs. An additional fragmentation header may be used to enable defragmenting the HT units by the receiver. HT units (e.g., small HT units) may be aggregated inside a single RTP PDU. An aggregation header may be used to enable retrieving individual HT units from an RTP PDU.

[0161] Feature(s) associated with haptics transport lEs are provided herein.

[0162] Haptics transport lEs may be used to indicate some characteristics of haptic data included in haptics media packets. Haptics transport lEs may be associated with a haptics media packet and / or with a haptics media flow. Haptics transport lEs may be used to improve the QoS of haptic services.

[0163] Per-packet haptics transport lEs may include one or more of the following: a sync type IE; a unit type IE; a dependency stamp IE; a dependency ID IE; a HTU layer IE; a transmission type IE; a haptic sequence number IE; or a haptics transport indication. Haptics transport indications may include one or more of the following: an end of haptic silence indication; a last fragment indication; a last dependent media packet indication; a sync indication; or an initialization indication.

[0164] A per-packet haptics transport IE (e.g., each per-packet haptics transport IE) may be associated with a haptics media packet (e.g., per-packet haptics transport lEs may be encoded in RTP header, RTP payload header, RTP fragment or aggregation header, MOQ or ROQ headers).

[0165] Per-packet haptics transport lEs may include a sync type IE (e.g., as represented by “D” in Table 4).

[0166] The sync type IE may be used to indicate whether an HTU included in the haptics media packet payload (e.g., RTP payload) is a sync HTU or not. A sync HTU may contain haptic data that may be decodedindependently from other HTUs, while a non-sync (e.g., dependent) HTU may require other HTUs to have been already decoded before the HTU itself may be decoded by the decoder. Loss of a sync HTU may not affect the end-user QoE more than the loss of a dependent HTU (e.g., since when losing an independent HTU, all dependent HTUs become unusable). In an example, an earlier dependent HTU may be more important to the QoE than a later dependent HTU (e.g., when both HTUs may depend on the same independent HTU). The sync type IE may be used to prioritize haptics media packets. In the case of transmission of aggregated HTUs in a haptics media packet, a single aggregated sync type IE may be present in the haptics media packet, for example with a sync value (e.g., an independent” value) if one or more sync HTUs are in the haptics media packet. A HTU in the haptics media packet (e.g., each HTU in the haptics media packet) may be associated with one per-HTU sync type IE. Using a sync type IE ( e.g., an aggregated sync type IE or per-HTU sync type IE), it may be possible to improve the QoS of a haptic data stream by prioritizing some haptics media packets (e.g., haptics media packets with sync type IE value sync) over other haptics media packets (e.g., haptics media packets with sync type IE value non-sync). The value of sync type may be represented by a single bit value, for example 0 and 1 . For example, in the case of a sync packet, the sync value may be marked as 0 (e.g., independent) or 1 (e.g., dependent).

[0167] Per-packet haptics transport lEs may include a unit type IE (e.g., as represented as UT in Table 4).

[0168] The unit type IE may indicate a HTU type. A user may define processing methods based on packet importance (e.g., based on a HTU type). A type of haptic media packet included may be checked based on a unit type IE. The importance of a haptic media packet may be analyzed to enable effective haptic media packet transmission.

[0169] The unit type IE may apply to HTUs, and encodings may be provided in a variety of ways. The IE may be encoded in a haptics media packet, which may contain more than one HTU. In the case of fragmentation, a haptic media packet may contain part of an HTU (e.g., fragmented). Several methods for marking packets to encode the unit type IE may be used.

[0170] An encoding example may be when a single packet or a fragmented packet may be included in an RTP packet. If there is a single HTU and / or a fragment of an HTU in a haptics media packet, the unit type IE value may be the value of the HTU unit type.

[0171] When more than one packet is included, the unit type IE may be expressed for each bit. The unit type IE may use 1 bit for each type, for example such that a summary may indicate at least one HTU of each type is present if the corresponding bit has a "1 " value.

[0172] When more than one packet is included, the unit type IE may be expressed for each bit implicitly (e.g., according to importance). If there are more than one HTUs in a haptics media packet, unit type IE value may be the value of the unit type of the most important HTU type in the packet. In examples, the importance of the HTU type may be determined by a pre-determined HTU type-to-importance mapping (e.g., initialization HTU may be more important than Temporal HTU which may be more important than Spatial HTU which may be more important than Silent HTU). In examples, the importance of the HTU type may be determined by a pre-determined mapping between the tuple (HTU type, HTU syn) and an importance (e.g., (Initialization, syn) > (Temporal syn) > (Silent, syn) > (Spatial syn) > (Temporal non-syn) > (Silent, non-sync)).

[0173] Per-packet haptics transport lEs may include a dependency stamp IE or dependency ID IE (e.g., as represented as DS in Table 4).

[0174] The dependency stamp IE of a non-sync HTU may indicate which sync HTU (e.g., a temporal-type HTU with sync attribute) the non-sync HTU depends on for decoding. The dependency stamp may be equivalently referred to as a dependency ID. The dependency stamp IE value may be used to detect the loss of a haptics media packet carrying a sync HTU. The dependency stamp IE value may be the same for all HTUs that depend on the same sync HTU. The dependency stamp IE value of two non-sync HTUs that depend on different sync HTUs may be different, for example if the two non-sync HTUs are close to each other in time. A dependency stamp IE may be associated with a haptics media packet (e.g., to indicate the dependency stamp of one or more HTUs carried in the haptics media packet). The dependency stamp IE may be used by a detection algorithm. For example, if a new sync haptics media packet has not arrived, but a receiving node (e.g., intermediate node or receiver node) observes new non-sync haptics media packets with a new dependency stamp IE value, it may indicate that a sync haptics media packet was lost in transmission. The receiving node may check whether a sync haptics media packet is missing, for example based on the value of the dependency stamp IE in multiple haptics media packets. If a new sync packet has not arrived but the dependency stamp IE value has changed in consecutive haptics media packets, this may indicate the loss of a sync haptics media packet. If a sync haptics media packet is detected as missing, the receiver may request a retransmission of the missing sync haptics media packet from the sender. The value of the dependency stamp IE may be encoded as an integer (e.g., using a limited number of bits), and therefore the dependency stamp IE value may roll over to zero periodically. A number of bits (e.g., 2 bits (for 4 possible dependency stamp values)) may be used. The receiver may determine how many haptics media packets are missing (e.g., using RTP packet sequence number) and use the detection algorithm described herein, for example when the number of lost packets is limited (e.g., 2 bits are used).

[0175] Per-packet haptics transport lEs may include a HTU Layer IE (e.g., as represented as L in Table 4).

[0176] HTU Layer IE may be used to indicate the order priority of HTU, for example as determined by the haptic sender (e.g., haptics codec). The order priority of HTU may be based on application-specific needs. The lower the HTU layer, the higher the priority that may be given to packet processing (e.g., by intermediate nodes and / or by receivers). For example, HTU Layer IE value may be used to determine the importance of a haptics media packet when network congestion occurs. In another example, when haptics media packets are transmitted along with other types of media packets, such as audio / video packets also associated with layer lEs, the haptics media packets’ HTU layer IE may be used with audio / video layer lEs to determine processing priority between the haptics / audio / video media packets. HTU layer values may be integers with, for example, 0 as the highest priority.

[0177] While the MIHS layer may be used by decoders to determine the importance of a MIHS packet for a decoding function, the HTU layer may be used by transport nodes to determine the importance of a haptics media packet for a transport function. For example, in a case where an HTU contains multiple MIHS packets, the HTU layer may be calculated as the minimum value of the MIHS layers of each MIHS packet in the HTU.

[0178] Per-packet haptics transport lEs may include a transmission type IE.

[0179] A transmission type IE may be used to represent the transmission type of haptic media packets. Transmission type may be used as a single HTU (e.g., a single MIHS packet mode). Transmission type may support an aggregation mode and / or a fragmentation mode.

[0180] A transmission type IE may support a single MIHS packet mode. For a single MIHS packet, all contents of MIHS may be marked in the IE (e.g., since one kind of packet may be included).

[0181] A transmission type IE may support an aggregation packet mode. For an aggregation mode, various types of MIHS may be included, and it may be possible to indicate the most important packet type (e.g., in the header) or to mention all packets (e.g., using additional bits).

[0182] A transmission type IE may support a fragment packet mode. For a fragment mode, one or two types of MIHS may be included, and it may be possible to indicate the most important packet type (e.g., in the header) or to mention all packets (e.g., using additional bits).

[0183] Each transmission mode may be encoded with a specific bit and / or may be expressed in combination with a unit type IE

[0184] Per-packet haptics transport lEs may include a haptic sequence number (HSM) IE (e.g., as represented as HSN in Table 4).

[0185] Each HTU may include haptic data covering a time duration (which may also be referred to as time slots or time frames). The time duration may be fixed (e.g., between an initialization and the next or the end of the stream) which, for example, may be provided by an initialization unit. For each consecutive and nonoverlapping constant time slots, zero, one, or more HTUs may be sent / received. The HSN IE may indicate the sequence number of the time slot corresponding to an HTU (e.g., a positive integer). The HSN IE may be zero for an initialization unit, and may be increased by one, for each consecutive time frame following the initialization unit. HSN IE may be reset as a 0 (e.g., when a new initialization unit starts).

[0186] The same HSN IE may be assigned to multiple HTUs covering the same time slot, which may be used to support multi-layer. For example, media packets carrying a first HTU with an HSN IE value of 12 may have an HTU layer IE value of 0 (base layer) and media packets carrying a second HTU with an HSN IE value of 12 may have an HTU layer IE value of 1 (upper layer).

[0187] A receiver may determine if there is a discontinuity in a stream of HTUs (e.g., Using the HSN IE), which may, for example, be caused by a media packet loss.

[0188] The receiver and / or an intermediate node may use the HSN IE to determine the priority of a media packet. For example, in case of congestion, if a node queues include packets with lEs (HSN=10,HTU-layer=0), (HSN=10,HTU-layer=1), (HSN=11 ,HTU-layer=1), (HSN=12,HTU-layer=0), the node may determine that the media packet with lEs (HSN=11 , HTU-layer=1) may be more important than other layer 1 packets (e.g., because it may contribute to a continuous experience to compensate for the loss of the layer 0 HTU in timeslot 11).

[0189] Per-packet haptics transport lEs may include haptics transport indication(s).

[0190] Haptic transport indications may be signaled in a haptics media packet header. For example, such indications may be signaled in the RTP header (e.g., using the Marker bit, or M-bit in the standard RTP header), or in a RTP payload header for haptics (e.g., using a bit dedicated for a specific indication). When information is represented using the RTP header, a packet may be processed without checking the payload of the packet (e.g., which may improve efficiency).

[0191] Haptics transport indications may include one or more of the following: an end of haptic silence indication; a last fragment indication; a last dependent media packet indication; a sync indication; or an initialization indication.

[0192] An end of haptic silence indication (e.g., the RTP header M-bit) may be used to mark the first haptics media packet (e.g., RTP packet) after a period of haptic silence.

[0193] Based on the end of haptic silence indication (e.g., RTP header marker bit indication), the receiver may be aware that this RTP packet is suitable for adapting the size / duration of the playout buffer (e.g., to improve playout synchronization with video and audio tracks, and / or to adapt to degraded network transmission performances), for example because a user may be less sensitive to a time shift after a period of silence, than during a continuous perception.

[0194] Based on the end of haptic silence indication (e.g., RTP header marker bit indication), the receiver may be aware that this RTP packet is suitable to trigger a “device washout” which may be an operation to reset the spatial position (e.g., “XYZ” axis) of a haptic actuator. A device washout may be performed during a period of silence (e.g., so it may be imperceptible for the end user). The receiver may use the reception of a silent HTU as a trigger for a device washout opportunity. The receiver may use any triggers (e.g., silent HTU, marker bit) along with other information, such as the time of the last device washout, to determine to perform a device washout operation.

[0195] A last fragment indication may indicate that the current packet may be the last media packet of a fragmented HTU. When an intermediate node or receiver node receives a fragmented haptic media packet, the node may process data without waiting for the packet (e.g., anymore).

[0196] A last dependent media packet indication may indicate that the current packet may be the last media packet of the last non-sync HTU (e.g., dependent HTU) preceding a sync HTU (e.g., independent HTU). When an intermediate node or a receiver node receives a haptic media packet including a last dependent media packet indication, it may prepare to receive the sync packet (e.g., partially re-initialize the decoder). If the next packet is not a sync, the intermediate node or the receiver node may act based on the configured policy (e.g., discard or wait for sync packet).

[0197] A sync indication may indicate that the media packet carries an HTU which is a sync (e.g., independent) HTU. A node may determine the priority of a haptic media packet based on a sync indication, for example without requiring parsing the RTP payload header for haptic. The media packet priority may be used for network congestion or loss recovery (e.g., as described herein).

[0198] An initialization indication may indicate the type of HTU is an initialization. A node may determine the priority of a haptic media packet based on an initialization indication, for example without checking the RTP payload header. The media packet priority may be used for network congestion or loss recovery (e.g., as described herein).

[0199] Feature(s) associated with a per-flow haptics transport IE are provided herein. A per-flow haptics transport IE may be associated with a haptics media flow and / or session (e.g., per-flow haptics transport lEs may be encoded in SDP messages or application messages). Per-flow haptics transport lEs may also be referred to herein as characteristics of a haptic stream and / or flow.

[0200] A sender may prepare a set of different streams corresponding to different capabilities of the possible receivers, for example based on input from haptics sensors and / or from an encoder. For example, if customers of an application provider own different sets of haptics devices (e.g., actuators), the sender may prepare haptic media streams that include haptic data for those different sets of devices. The sender may associate per-flow haptics transport lEs to each media stream. The sender may provide a list of media streams descriptions, including per-flow haptics transport lEs, to a receiver. The receiver may, based on the characteristics of the haptics devices used to play the received haptics data and / or based on the per-flow haptics transport lEs provided by the sender, select the media stream(s) it wishes to receive. The receiver may, in addition or replacement of selecting media stream(s), select a new value for a per-flow haptics transport IE and associate it with a media stream description (e.g., which the receiver may send to the sender).

[0201] This behavior by the receiver may enable one or more of the following: identification of a stream that is played by the haptics devices at the receiver; help limiting bandwidth usage; or help limit the impact of packet losses and congestion on the QoE for the end user.

[0202] The receiver may provide the selected media stream(s) to the sender (e.g., in an SDP answer). The sender may stream the selected media stream(s) to the receiver. The receiver may receive, decode, and / or provide the haptics media data to the haptics device(s), which may render the haptics data to the user. In examples, the order of messages may differ (e.g., the receiver may send haptics media description with per- flow transport lEs to the sender). The sender may select media flow(s) to send based in part (e.g., at least in part) on per-flow transport lEs.

[0203] Per-flow haptics transport IE may include one or more of: a preferred haptic device frequency ranges IE; a supported haptic device frequency ranges IE; a perception modality IE; a type of actuator IE; a layer information IE; a haptics encoding method IE; a body part mask IE; a haptic silence suppression IE; a default haptics signal IE; a version IE; a profile IE; a level IE; a layer vs. sync priority IE; a media type priority IE; a haptics congestion control configuration IE; or a selective haptics loss recovery configuration IE.

[0204] Per-flow haptics transport IE may include a preferred haptic device frequency ranges IE and / or a supported haptic device frequency ranges IE.

[0205] Haptics devices (e.g., sensors or actuators) may generate or may play haptics data (e.g., within a frequency range). Some haptic devices may more efficiently play haptics data within particular frequency range(s) and may play haptics data within other frequency range(s) less efficiently (e.g., in a way less important to the end user).

[0206] A receiver may be connected and / or attached to player haptics device(s) (e.g., actuators). These devices may be referred to as the receiver’s devices.

[0207] A supported haptic device frequency ranges IE may indicate one or more frequency ranges that are supported by a node. A sender may set a supported haptic device frequency ranges IE to indicate the frequency range(s) that may be present in a data stream. A receiver may set a supported haptic device frequency ranges IE to indicate the frequency range(s) that the haptic devices (e.g., that it is connected to) support.

[0208] A preferred haptic device frequency ranges IE may indicate one or more frequency ranges that are preferred by a node (e.g., by a sender and / or receiver). A sender may set a preferred haptic device frequency ranges IE to indicate the preferred frequency range(s) (e.g., most efficient for QoE) that may be generated by the source haptic devices and / or by the encoder. A receiver may set a preferred haptic device frequency ranges IE to indicate the preferred frequency range(s) (e.g., mostly efficient for the QoE of the end user) for the haptic player devices that may be connected to the receiver.

[0209] In examples, a sender may query the haptic device frequency range(s) that are supported by source haptic devices and / or by another source, such as a haptics media file. The sender may send a supported haptic device frequency ranges IE that may indicate the queried frequency range(s) associated with a media flow description. The sender may send multiple media flow descriptions with different supported haptic device frequency ranges lEs (e.g., to support different sets of receivers that use different sets of player haptics devices, for example a stream for a high-end haptics suit may support a large frequency range and another stream for a helmet-and-gloves setup mat support a narrow frequency range).

[0210] The receiver may compare the supported haptic device frequency ranges lEs from the sender to the frequency ranges that are supported by the player haptics devices attached / connected to the receiver. The receiver may select a haptics media stream that is associated with supported haptic device frequency ranges that are compatible (e.g., playable) with the receiver’s devices. The receiver may respond to the sender with the selected media stream(s) (e.g., by including them in an SDP answer message).

[0211] In some cases, the receiver may determine a preferred haptic device frequency range IE associated with a selected media stream (e.g., before sending back a response to the sender). For example, the receiver may be configured by the application provider to prefer some types of haptic devices for rendering. For example, a client application may configure the receiver to give more importance to certain types of actuators (e.g., gloves) than other types of actuators (e.g., helmets). The receiver may look up the capabilities of the preferred haptics devices and may use their frequency ranges as preferred frequency ranges. The receiver may set a preferred haptic device frequency ranges IE with these value(s) and may associate this IE with the answer it may send to the sender (e.g., by including the preferred haptic device frequency ranges IE within the media description of the selected haptic media stream(s)). The receiver may select and / or send a preferred haptic device frequency range. The preferred haptic device frequency range may cover all or some of the frequency ranges supported by all of its devices based on user or application policy (e.g., purpose of use, frequency of use).

[0212] When the sender receives a selected haptics media stream with a preferred haptic device frequency ranges IE, the sender may update the haptics data (e.g., before or while streaming). For example, the sender may set the HTU layer to 0 (e.g., base layer, most important) for HTUs that contain haptics data containing frequencies within the preferred ranges. The sender may set the HTU layer to 1 or more (e.g., less important than HTU layer 0) for HTUs that contain haptics data not containing frequencies within the preferred ranges.

[0213] Using supported haptic device frequency ranges lEs may enable a suitable choice of media streams by the receiver (e.g., adapted to the receiver’s haptic devices, and / or network efficient).

[0214] Using preferred haptic device frequency ranges lEs may enable a proper marking of haptics media packets by the sender (e.g., by setting HTU layer IE), that may enable network nodes to properly prioritize haptics media packets (e.g., to reduce the impact of network congestion on end user QoE).

[0215] Per-flow haptics transport IE may include a perception modality IE. A perception modality IE indicates one or more types of perceptions (e.g., pressure, acceleration vibrotactile) that are supported by node. A sender may set this IE to indicate the type of perception that may be present in a data stream. A receiver may set this IE to indicate the type of perception that the haptic devices (e.g., the haptic devices that it is connected to) support.

[0216] In examples, a sender may query the perception modality the haptic device has that are supported by source haptic devices and / or by another source such as a haptics media file. The sender may send a supported haptic device perception modality IE that may indicate the queried type of perception modality (e.g., associated with a media flow description). The sender may send multiple media flow descriptions with differentsupported type(s) of perception modality lEs (e.g., to support different sets of receivers that use different sets of player haptics devices, for example a combination of various haptic media players).

[0217] The receiver may compare the supported perception modality lEs from the sender to the perception modality that are supported by the player haptics devices attached / connected to the receiver. The receiver may select a haptics media stream that is associated with supported perception modality that may be compatible (e.g., playable) by the receiver’s devices. The receiver may respond to the sender with the selected media stream(s) (e.g., by including them in an SDP answer message).

[0218] In some cases, the receiver may determine a type of perception modality IE associated with a selected media stream (e.g., before sending back a response to the sender). For example, the receiver may prioritize different types of perception modalities and / or may configure a set of perception modalities selected by the application. The receiver may set the type of perception modality or set of perception modality IE with these value(s) and / or may associate this IE with the answer the receiver sends to the sender.

[0219] When the sender receives a selected haptics media stream with a specific perception modality IE, the sender may update the haptics data (e.g., before or while streaming). For example, the sender may, for each time period, packetize preferred perception modality data in one packet and other perception modalities in other packets (e.g., the sender may set the HTU layer to 0 for packets including a preferred perception modality, and HTU layer 1 for other packets). When a (pre)defined set of perception modality available at the sender changes, the haptic stream may be updated by renegotiating the per-flow haptic transport lEs between sender and receiver. The sender may (e.g., alternatively) prioritize individual perception modalities by organizing them into different flows.

[0220] The haptic effect may be provided (e.g., selectively provided) depending on whether the receiver may provide perception modality and / or the type of haptic media of the sender. Using perception modality IE, the effective data transmission may be possible even in a network congestion situation (e.g., by selectively utilizing the preferred effect). For example, in a network congestion situation, only important perception modality (e.g., main perception device data) may be forwarded (e.g., selective forwarding).

[0221] Per-flow haptics transport IE may include a type of actuator IE. A type of actuator IE may indicate one or more type of actuators (e.g., LRA, VCA, ERM) that are supported by node. A sender may set this IE to indicate the type of actuator that may be present in a data stream. A receiver may set this IE to indicate the type of actuator that the haptic devices (e.g., the haptic devices that it is connected to) support.

[0222] In an example, a sender may query the type of actuators that are supported by the haptic source (e.g., a haptic sensor, haptics media file or haptic stream source). The sender may send a supported haptic device type of actuator IE that may indicate the queried type of actuators (e.g., associated with a media flow description). The sender may send multiple media flow descriptions with different supported type of actuator lEs (e.g., to support different sets of receivers that use different sets of player haptics devices, for example a combination of various haptic actuators).

[0223] The receiver may compare the supported type of actuator lEs from the sender to the type of actuators that may be supported by the player haptics devices attached / connected to the receiver. The receiver may select a haptics media stream that is associated with a supported type of actuator that is compatible (e.g., playable) by the receiver’s devices. The receiver may respond to the sender with the selected media stream(s) (e.g., by including them in an SDP answer message).

[0224] In examples, the receiver may determine a type of actuator IE associated with a selected media stream (e.g., before sending back a response to the sender). For example, the receiver may prioritize different types of actuators and / or may configure a set of actuators selected by the application. The receiver may set a type of actuator IE with these value(s) and / or may associate this IE with the answer it may send to the sender.

[0225] When the sender receives a selected haptics media stream with a specific type of actuator IE, the sender may update the haptics data (e.g., before or while streaming). For example, when a predefined set of actuators or single actuator changes, it may be updated. The sender may prioritize each actuator by organizing the haptics data into different flows.

[0226] Since the power, volts, and / or waveform used may vary (e.g., depending on the type of actuator), overall communication efficiency may be increased by selectively using the actuator (e.g., a specific type of actuator).

[0227] The haptic effect may be selectively provided depending on whether the receiver provides actuators and the type of haptic media of the sender. Using type of actuator IE, effective data transmission may be possible even in a network congestion situation (e.g., by selectively utilizing the preferred effect).

[0228] Per-flow haptics transport IE may include a layer information IE.

[0229] An IE (e.g., each IE) may be configured as a single flow, and multiple lEs may be configured as a single flow when configured as a set. A flow (e.g., each flow) may correspond to one layer. A flow (e.g., each flow) may be used selectively, for example by defining priority to each layer.

[0230] A layer information IE (e.g., layer IE) may indicate one or more layers (e.g., HTU layers) that may be supported by a node. A sender may set a layer information IE to indicate a layer that may be present in a data stream. A receiver may set a layer information IE to indicate the layer that the network (e.g., a mobile access network) and / or the haptic devices (e.g., that the receiver is connected to) support.

[0231] In an example, a sender may query a layer that is supported by source haptic devices (or by another source such as a haptics media file). The sender may send a layer information IE that may indicate the queried layer (e.g., associated with a media flow description).

[0232] For example, the priority of a flow that includes various information may be allocated by using a layer. When the sender and receiver have multiple flows for one IE, it may be prioritized through layer configurations.

[0233] This process may be negotiated with the same procedure as the existing negotiation process after the flow for IE is determined. For example, the flow generated for an IE (e.g., each IE) may be assigned to a specific layer to give priority to desired negotiation information.

[0234] The receiver may compare the supported layer information IE(s) from the sender to the layer information that may be supported by the player haptics devices attached / connected to the receiver. The receiver may select a haptics media stream that is associated with supported layer info that is compatible (e.g., playable) by the receiver’s devices. Layer information lEs may be useful for selecting high priority or preferred data (e.g., layers 0 to N where N may be determined by the receiver based on its devices, application, and / or access network level of service) over lower priority or less preferred data (e.g., layers N+1 and higher). The receiver may respond to the sender with the selected media stream(s) (e.g., by including in the media stream only the layers requested by the receiver).

[0235] The receiver may determine a layer information IE associated with a selected media stream (e.g., before sending back a response to the sender). For example, the receiver may prioritize different layers and / or may configure a layer selected by the application. The receiver may set a layer information IE with these value(s) and / or may associate a layer information IE with the answer it may send to the sender.

[0236] When the sender receives a selected haptics media stream with a specific type of layer IE, the sender may update the haptics data (e.g., before or while streaming). For example, when a predefined layer changes, it may be updated (e.g., deleted or added). The sender may prioritize layers (e.g., each layer) by organizing them into different flows.

[0237] The receiver and sender may negotiate layers according to the layer’s priorities and / or may selectively provide haptic effects. Using Layer IE, priority may be assigned to various lEs and data may be transmitted effectively according to network conditions.

[0238] Per-flow haptics transport IE may include a haptics encoding method IE. A haptics encoding method IE may indicate one or more haptics encoding methods (e.g., vectorial, wavelet) that are supported by a node. A sender may set a haptics encoding method IE to indicate haptics encoding method(s) that may be present in a data stream. A receiver may set a haptics encoding method IE to indicate the haptics encoding method(s) that the haptic devices (e.g., the haptic devices that it is connected to) support.

[0239] A sender may query haptics encoding method(s) that are supported by source haptic devices and / or by another source, such as a haptics media file. The sender may send a supported haptic device haptics encoding method IE that may indicate the queried haptics encoding method(s) (e.g., associated with a media flow description). The sender may send multiple media flow descriptions with different supported haptics encoding method lEs (e.g., to support different sets of receivers that use different sets of haptics devices).

[0240] The receiver may compare the supported haptics encoding method lEs from the sender to the haptics encoding method(s) that are supported by the haptics devices attached / connected to the receiver. The receiver may select a haptics media stream that is associated with the supported haptics encoding method(s) that are compatible (e.g., playable) by the receiver’s devices. The receiver may respond to the sender with the selected media stream(s) (e.g., by including them in an SDP answer message).

[0241] The receiver may determine a haptics encoding method IE associated with a selected media stream (e.g., before sending back a response to the sender). For example, the receiver may prioritize different haptics encoding method(s) or may configure a haptics encoding method selected by the application. The receiver may set a haptics encoding method IE with these value(s) and / or may associate a haptics encoding method IE with the answer it may send to the sender.

[0242] When the sender receives a selected haptics media stream without haptics-encoding-method IE, the sender may start streaming the selected media stream.

[0243] When the sender receives a selected haptics media stream with a specific haptics encoding method IE, the sender may update the haptics data (e.g., before or while streaming). For example, when a haptics encoding method changes, it may be updated. The sender may prioritize haptics encoding method(s) (e.g., each haptics encoding method) by organizing them into different flows.

[0244] By selecting an appropriate encoding method for the transmission, an optimized data stream method may be provided. Selecting an appropriate encoding method for the transmission may reduce encoding costs when common encoding method(s) are used for various devices.

[0245] Per-flow haptics transport IE may include a body part mask IE. A body part mask IE may indicate (e.g., where the haptic effect may be played) that haptic devices for one or more body parts are supported by a node. A sender may set this IE to indicate the body parts that have corresponding data present in a data stream. A receiver may set this IE to indicate the body parts that the haptic devices support (e.g., the parts of the body that are served by those devices).

[0246] A sender may query body part mask(s) that are supported by source haptic devices and / or by another source, such as a haptics media file. The sender may send a supported haptic device body part mask IE that may indicate the queried body part mask (e.g., associated with a media flow description). The sender may send multiple media flow descriptions with different supported body part mask lEs (e.g., to support different sets of receivers that use different sets of player haptics devices).

[0247] The receiver may compare the supported body part mask lEs from the sender to the body part mask(s) that are supported by the haptics devices attached / connected to the receiver. The receiver may select a haptics media stream that is associated with supported body part mask(s) that are compatible (e.g., playable) by the receiver’s devices. The receiver may respond to the sender with the selected media stream(s) (e.g., by including them in an SDP answer message).

[0248] The receiver may determine a body part mask IE associated with a selected media stream (e.g., before sending back a response to the sender). For example, the receiver may prioritize different body parts or may configure a body part mask selected by the application. The receiver may set a haptics encoding method IE with these value(s) and / or may associate this IE with the answer it may send to the sender.

[0249] When the sender receives a selected haptics media stream with a specific body part mask IE, the sender may update the haptics data (e.g., before or while streaming). For example, when a body part mask changes, it may be updated (e.g., deleted or added). The sender may prioritize each haptics encoding method by organizing them into different flows.

[0250] It may be possible to assign priorities to various flows, set parameters optimized for the data transmission environment, and transmit data by utilizing the body part mask. For example, data may be transmitted for all body parts when the network condition is good and for select part(s) of the body where it may represent the effect well when the network condition is bad (e.g., selective transmission).

[0251] Other lEs may be used to enable adaption (e.g., selection or marking) of haptics data by the media sender, for example based on a negotiation between media sender and receiver. Other lEs may include one or more of the following: a date IE, which may indicate the creation date of the haptic experience; a description IE, which may indicate a user-defined description of this haptic experience; a timescale IE, which may indicate a number of ticks per second; an avatar level of detail IE, which may indicate which levels of details to use for the avatar(s); an avatar type IE, which may indicate the type of haptic perception represented by the avatar(s); a resonance frequency IE, which may indicate the resonance frequency of the actuators (Hz); a maximum amplitude IE, which may indicate the maximum amplitude value of the targeted device; an impedance number IE, which may indicate the impedance of the actuator(s); a maximum voltage IE, which may indicate the maximum voltage of the actuator; a maximum current IE, which may indicate the maximum current of the actuators; a maximum displacement IE, which may indicate the maximum displacement of the actuator; a device weight IE, which may indicate the weight of the device; a device size IE, which may indicate the size of the device; a custom device IE, which may indicate user-defined data; or a device type IE, which may indicate the type of actuator(s).

[0252] Per-flow haptics transport IE may include a haptic silence suppression IE, which may indicate whether haptic silence suppression is supported by a media sender and / or receiver.

[0253] A sender may set the haptic silence suppression IE (e.g., to a boolean value “true” or an integer value N) to indicate that it has the capability to not send some silent MIHS units during a period of haptic silence (e.g., to limit network resource usage). A receiver may set the haptic silence suppression IE to indicate that it supports interpreting a time period without MIHS units as a period of silence, and / or that the receiver will continue to operate the decoder with implicit silent MIHS units during this period.

[0254] In an exemplary implementation of haptic silence suppression, the sender may obtain silent HTUs (e.g., silent MIHS units) from the encoder and / or from a file or other haptic media source that may indicate there is no haptic effect to render during a time period. The sender may send the first N silent HTUs (e.g., which may, in some systems, be the value of the haptic silence suppression IE). For example, N may be set to 2 or 3 to increase the probability that the receiver will receive at least one silent HTU (e.g., even over a lossy network). The sender may stop sending any following silent HTUs until a non-silent HTU is available to send. The receiver may receive the initial silent HTUs and may interpret the following missing HTUs as silent (e.g., with high certainty).

[0255] The sender may set an indication in the first non-silent haptic media packet after a period of silence, (e.g., by setting the RTP header marker bit, for example as described herein).

[0256] Per-flow haptics transport IE may include a default haptics signal IE that may indicate replacement haptic data (e.g., haptic effects) that should be played during a haptic silence. The default haptics signal IE value may be a set of effects and / or may be a reference to haptic effects present in the haptic stream. For example, the default haptics signal IE may be a reference to effect(s) in an effect library, for example that may be communicated in an initialization HTU.

[0257] A sender may set the default haptics signal IE to application specific effect(s) suitable to be played when there is no received haptic signal. For example, during an immersive application where the user is in a virtual environment with background vibrations, the sender may set the default haptics signal IE to intermittent background vibrations (e.g., to better simulate a real environment).

[0258] Per-flow haptics transport IE may include a version IE that may indicate the version of the haptic specifications used for transporting the haptics data flow.

[0259] Per-flow haptics transport IE may include a profile IE that may indicate a set of encoding and decoding tools used to generate and consume the haptics data stream. For example, a profile IE value may indicate a reduced set of tools, therefore enabling simpler implementations adapted to the capability of the haptic devices used.

[0260] Per-flow haptics transport IE may include a level IE that may indicate a further refinement of the profile IE. For example, a profile may be defined with two levels. In examples, a representative device of level 1 may be mobile device (e.g., actuator) and a representative device of level 2 may be a haptic suit (e.g., configured to support multiple haptic channels).

[0261] Per-flow haptics transport IE may include a layer vs. sync priority IE. When a layer vs. sync priority IE value is “layer” the IE may indicate that a haptic media packet with lower layer may have a higher priority than a haptic media packet with a higher layer (e.g., regardless of its sync value). When a layer vs. sync priority IE value is “sync” the IE may indicate that a sync haptics media packet may have a higher priority than a non-sync lower layer packet (e.g., regardless of its layer).

[0262] Per-flow haptics transport IE may include a media type priority IE that may indicate the relative priority between related audio, video, and / or haptics flows. For example, if haptic is more important than audio, a media type priority IE may indicate to give more priority to the haptic media flow over the audio media flow.

[0263] Per-flow haptics transport IE may include a haptics congestion control configuration IE that may indicate whether a congestion control algorithm for haptics data (e.g., as described herein) should be used in case of a congestion event.

[0264] Per-flow haptics transport IE may include a selective haptics loss recovery configuration IE that may indicate whether a haptics loss recovery algorithm (e.g., as described herein) should be used to determine which packet should be re-transmitted.

[0265] Feature(s) associated with IE transmission methods are provided herein.

[0266] Per-packet metadata may be encoded as bit fields in the RTP Payload Header and / or in RTP payload fragmentation headers and / or in RTP payload aggregation header. Per-packet metadata may also be in other media transport headers, e.g., MOQ or ROQ header, etc.

[0267] SDP and / or the application layer will be used to exchange session information between endpoints before starting the haptic data exchange.

[0268] Haptic media packets may be transmitted as different forms (e.g., according to the transmission type). For example, haptic media packets may be transmitted in three forms: a single mode, a fragmentation mode, and an aggregation mode. Examples of per-packet haptics transport lEs encoding in RTP, and other transport protocols, are provided herein.

[0269] FIG. 6 illustrates example transmission modes in RTP.

[0270] As shown in FIG. 6, haptic media data may be transmitted with 3 modes. Haptic media data may be transmitted with a single HTU packet, with an aggregated RTP packet (e.g., when haptic media packet size is small), or with a fragmented RTP (e.g., when the packet size is bigger than RTP payload size).

[0271] The haptics media flow may be transported using different methods, including using encrypted transport methods such as RTP over QUIC and / or Media over QUIC. The haptics transport lEs may be transported in header fields that that may be made available to intermediate nodes (e.g., along with other MOQ or ROQ transport metadata, such as object ID, or track information), for example rather than being encrypted end-to-end. For example, haptics transport lEs may be transported in hop-by-hop MOQ or ROQ headers or messages. To access haptics transport lEs encoded in such a manner, an intermediate node may be an ROQ or MOQ proxy. MOQ and ROQ proxies may access MOQ / ROQ transport metadata.

[0272] Table 1 depicts an example single time aggregated RTP packet structure.Table 1: example single-time aggregated RTP packet structure0 1 2 30 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1| RTP Payload Header | HTU Header |

[0273] A HTU header may have a minimum of 1 byte and may be used up to 2 bytes. The packet structure in an aggregated packet may be decided according to the size of the packet header or packet data. An AU Header may comprise a HTU Size and HTU Header bit. Table 2 shows a single-time aggregation packet(STAP) structure, which may be used in haptics to transmit multiple packets using the same timestamp. For example, if two frequencies are used for the same content, they may be transmitted at once through STAP.

[0274] Table 2 depicts an example multi-time aggregated RTP packet structure.Table 2: example multi-time aggregated RTP packet structure.0 1 2 30 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1| RTP Payload Header | HTU Header || RTP Payload Header | HTU Header |

[0275] The packet structure of multi-time aggregation may be used when transmitting multiple MIHS units with different time stamps in haptics. For example, various MIHS units corresponding to time order may be included in one packet.

[0276] Table 3 depicts an example structure of a fragment RTP packet. FUS may refer to fragmented unit start and FUE may refer to fragmented unit end. FUS and FUE may be used to mark the period of fragmentation.Table 3: example transmission modes in RTP.

[0277] In RTP packets the header may be defined as shown in Table 4 (e.g., to provide information on haptic data included in the RTP payload and to efficiently transmit data in network congestion / packet loss situations). F may refer to a forbidden bit, D may refer to a sync type bit, UT may refer to a unit type bit, DS may refer to a dependency stamp, L may refer to a HTU layer, and RSV may refer to a reserved bit. RSV may be used for additional information. According to the usage, RTP packets may have different (e.g., two) kinds of payload headers.Table 4: examples of TRP payload header definitionExample 1 : 1 byte header Example 2 : 2 bytes header

[0278] All lEs may be encoded (e.g., in various methods). There may be no restriction on the encoding methods. For lEs containing similar contents, it may be possible to use a unified encoding method by utilizing bits in one IE for efficient encoding.

[0279] For example, the unit type value may be expressed in aggregate with the transmission type value (e.g., integrated). When the MIHS unit type is represented with a specific value, aggregation mode and fragmentation mode may be assigned to the specific value and represented together.

[0280] Example unit type values may include one or more of the following: 0 = Initialization (single); 1 = temporal (single); 2 = spatial (single); 3 = silent (single); 14 = Single-Time Aggregation; 15 = Multi-Time Aggregation; or 16 = fragmentation.

[0281] Additional headers may be added (e.g., depending on the packet transmission method). Information from the lowest header (e.g., inner) may be reflected in the upper header. For example, if the lower data is a sync packet, the importance may be emphasized by expressing this information in the upper header as well.

[0282] Information in haptic media data may be expressed using RTP payload header(s), but some information may be signaled in the RTP header. If the haptic media data information is signaled using the RTP header, it may be possible to process the packet without checking the inner packet. For example, a new sync cycle or packet type of Haptic media data may be mentioned by using a specific bit in the RTP header.

[0283] A bit in the RTP Payload header may be used to encode a haptic transport indication. For example, the RTP header marker bit (M-bit) may be used to encode a haptic transport indication.

[0284] New attributes may be defined to encode each of the per-flow haptics transport lEs defined herein (e.g., to negotiate haptic data communications). Below is an example of a haptic media description extending the SDP format to describe a haptics stream and its associated attributes:1) m=haptics 43291 UDP / TLS / RTP / SAVPF 1152) c=IN IP4 192.168.2.83) b=AS:644) a=mid:streamO5) a=recvonly6) a=ssrc:1 cname:streamO7) a=rtcp-mux8) a=rtpmap: 115 hmpg / 80009) a=fmtp:115 profile-id=1 ;level-id=1 ;version=2023; haptic-silence-suppression

[0285] Line 1 may indicate that the type of media is haptic, and the numbers following it may indicate the port used. '115' may refer to the payload type. Lines 2 to 7 are SDP lines using exemplary values. The rtpmap in line 8 may map the payload type (e.g., 115), with a media format (e.g., HMPG, designating here a new haptics MPEG media codec), and a sampling rate (e.g., 8000 for 8kHz). Line 9 may provide the value of new attributes specific to the new haptics format. Attributes may be defined for any per-flow haptics transport IE defined herein. This example uses four exemplary attributes, based on per-flow haptics transport lEs version, profile, level, and haptic silence suppression.

[0286] Identifying and / or marking the most important portion of the haptics traffic may be performed.

[0287] In a haptic environment, there may be various types of devices (e.g., sensors and actuators), and the data type or capabilities provided or consumed by each device may be different. Some device types may contribute more to the user QoE than others. Some haptic data frequency ranges may contribute more than other ranges to the user QoE (e.g., for some devices). To transmit haptic media data efficiently, it may be useful to reduce network and processing resource usage, to share information between the sender and receiver, and / or to configure the transmission environment before transmitting data with per-flow haptic transport lEs. For example, the sender and receiver may use a media session negotiation procedure for exchanging the information (e.g., using an SDP session establishment or other media session negotiation procedure). Based on the result of the negotiation, the sender may select the appropriate haptic stream(s) to send (e.g., if multiple haptic streams are used to send separate modalities, the sender may select the haptic streams requested by the receiver). Based on the result of the negotiation, the sender may mark the haptic media packets (e.g., with an HTU layer value), to indicate the importance or priority of the haptic data in the HTU (e.g., HTU layer 0 for most important data, layer 1 for less important, 2 for least important to the receiver). Based on the result of the negotiation, the sender may packetize the haptic media packets. A basic packetization may make one HTU per time period and / or may place all haptic data in the HTU. To support differentiating a preferred portion of the haptic data (e.g., a perception, a modality, a frequency band), another type of packetization may make several HTUs per time period, and / or may set the HTU layer of each HTU to different values. Most important haptic data may be placed in the HTU with HTU layer 0, less important haptic data may be placed in the HTU with HTU layer 1, etc.

[0288] In the media session negotiation procedure, sender and receiver may exchange per-flow haptic transport lEs. Per-flow haptic transport lEs may be generated and used based on application logic or local enduser devices. The procedure of exchanging information may be initiated by either the sender or the receiver.

[0289] In examples one or more of the following may be performed for an initiation. The sender may transmit a list of characteristics of haptic streams (e.g., identified using Per-flow haptic transport lEs) that may be provided to the receiver (e.g., based on the available data sources and their capabilities). For example, if there are two types of modalities (e.g., pressure, temperature) that the sender may provide, the sender may advertise this information to the receiver. After receiving the information, the receiver may (e.g., based on the haptic devices available at the receiver or based on the end-user preferences) send a response message to the sender only including the requested (e.g., desired, or supported) characteristic among the available options. If the receiver requires both modalities, the sender may send haptic data for all modalities. If the receiver requires only one modality, the sender may send haptic data for only the selected modality. If thereceiver requires both modalities but prefers the first modality (e.g., because it impacts end-user QoE more than the second), the sender may send both modalities, and mark haptic packets holding haptic data for the first modality as the base layer (e.g., as a high priority), and the other modality with a higher layer (e.g., as a low priority). This may enable the network and endpoints to apply congestion control or packet loss recovery algorithms (e.g., giving priority to the preferred modality).

[0290] Multiple per-flow haptic transport lEs are defined herein, and between sender and receiver each of these IE may be configured for multiple flows simultaneously. In addition, the sender may change (e.g., modify, add, and / or delete) a haptic flow depending on the situation. For example, if the sender provides two kinds of bands for haptic data and only one band flow can be delivered due to a device problem or changing network conditions, the sender may control the flow by changing the priority of the flow and / or deleting unnecessary bands.

[0291] In cases where the receiver initiates the media session negotiation, the receiver may use per-flow haptic transport lEs to share the capabilities of its own device, as well as user or application preferences with the sender, and / or request supported / preferred haptic data stream characteristics from the sender. The sender and receiver may renegotiate based on triggers originating from the end-user (e.g., an end-user changes the application configuration to add or remove devices / modalities), from the receiver (e.g., a device fails to function properly), from the sender (e.g., a new stream or modality becomes available), and / or from the network (e.g., the network may send a congestion notification, for example, using ECN, to an endpoint, which may trigger the endpoint to the amount of haptic data it may receive). Such re-negotiation may result in a change in the composition of the haptic stream, and / or in how the haptic stream is packetized and marked (e.g., a transition from sending all haptic data for a time period in the same HTU, to splitting haptic data across two different HTUs per time period, for example, one HTU layer 0 for a more important modality and another HTU layer 1 for the other modalities). Other examples of marking (e.g., besides HTU layer) may include setting PDU set importance, for example, by the media sender (e.g., in an RTP header extension for PDU set handling).

[0292] Feature(s) associated with identifying and / or marking an important (e.g., most important) portion of haptics traffic are provided herein.

[0293] Information may be shared between the media sender and the media receiver and / or the transmission environment may be configured before transmitting data with per-flow haptic transport lEs (e.g., SDP session establishment).

[0294] FIGS. 7A and 7B depict an example haptics traffic negotiation (e.g., during network congestion).

[0295] As illustrated in FIGS. 7A and 7B, a first exemplary negotiation may occur at 0-2b.

[0296] At 0, the media receiver may send a streaming request (e.g., an application layer message) to express the request of a user to start a media stream.

[0297] At 0a, the media sender may determine a range of characteristics of the streams it may send to the media receiver (e.g., based on application logic). The media sender may determine the methods that may be used to identify portions of the haptics stream (e.g., per perception, per frequency band, etc.) and associated parameters (e.g., list of perceptions available, list of frequency bands available, etc.), for example based on the determined characteristics.

[0298] At 1, the media sender may send an offer message to the media receiver, (e.g., including per-flow haptic transport lEs) that may include a description of the different methods that may be used to identify portions of the haptics stream that can be transmitted with different quality of service, and associated parameters.

[0299] At 1a, the media receiver may determine user configuration, user input, and / or locally available haptic devices (e.g., based on application logic). The media receiver may determine supported and / or preferred characteristics of the stream.

[0300] At 2, the media receiver may send (e.g., to the media sender) an answer message (e.g., including per-flow haptics transport lEs) that may indicate the methods and associated parameters that are selected by the media receiver.

[0301] At 2a, the media sender may select the haptic stream(s) to send based on the method and associated parameters selected by the media receiver. The media sender may stream the haptics media stream, packetizing and / or marking media packets it may send as per the method and associated parameters selected by the media receiver (e.g., at 2b).

[0302] As illustrated in FIGS. 7A and 7B, a second exemplary negotiation may occur at 3a-4b.

[0303] At 3a, the media receiver may request more data (e.g., add more frequency bands to improve the experience).

[0304] At 3b, the media receiver may send an offer message to the media sender. The message may include per-flow haptics transport lEs that may indicate the methods and associated parameters. For example, the offer may be based on the previous answer and / or may associate new frequency bands with a haptic stream.

[0305] At 3c, the media sender may determine the updated stream characteristics. For example, the media sender may ensure that the requested new frequency bands are available. The media sender may accept the requested changes as is, reject them (e.g., keep the original frequency bands), or modify them (e.g., change the frequency bands to the intersection between the requested new frequency bands and what the media sender has the capability to send).

[0306] At 4, the media sender may send an answer message to the media receiver that may indicate the accepted characteristics of the stream. Upon reception at 4a, the media receiver may prepare itself to receive the updated stream (e.g., if needed). At 4b, the media receiver may start to transmit the updated stream.

[0307] As illustrated in FIGS. 7A and 7B, a third exemplary negotiation may occur at 5a-6a.

[0308] At 5a, the media sender may determine to update the haptic flow (e.g., because the network environment has changed, or the media source doesn’t support specific frequency bands anymore).

[0309] At 5b, the media sender may trigger a re-negotiation procedure by, for example, sending an offer message. For example, the media sender may base the answer on a previously accepted answer and / or may change some characteristics of the stream (e.g., remove, add, or change the priority of some portion of the haptic stream).

[0310] At 5c and 6, the media receiver may prepare an answer that accepts the changes from the offer and / or updates the offer (e.g., the media receiver may prefer to stop the stream rather than accept the changes). The media receiver may send the answer to the media sender.

[0311] At 6a, the media sender may implement the changes (e.g., by changing the selection of the haptic stream to send, changing the packetization, or marking of haptic media packets it may send to the media receiver).

[0312] Feature(s) associated with congestion control for haptic data are provided herein.

[0313] Network congestion may occur when transmitting data in the internet environment or over a wireless link. A QoE-efficient method to effectively respond to network congestion may be considered (e.g., based on the haptic media packet). The haptics transport IE defined above may be used to determine which haptic media packets to drop in network congestion situations.

[0314] Congestion (e.g., and the congestion control algorithm described herein) may be experienced by a media sender or by an intermediate node in the network. In a media sender, congestion may be detected when the sending queue length increases when using a media congestion control-based transport protocol (e.g., when using a protocol such as MOQ to transport media), or congestion may be detected using a controlmessage from an endpoint or network (e.g., an RTCP message may indicate the non-reception of some RTP packets). In an intermediate node (such as an RTP middlebox, or a UPF), received haptics media packets may be queued for transmission towards the receiver node or next hop towards the receiver node. For example, upon arrival the haptics media packets may be queued at the back of the queue, and when there is an opportunity for transmission, a haptics media packet may be dequeued at the front and transmitted. This may ensure that the haptics media packets are transmitted without reordering packets. In an intermediate node, congestion may be detected when the transmission queue grows above a threshold (e.g., a time duration, or number of packets) based on configuration (e.g., from the network operator). In a media sender and an intermediate node, dropping packets may be necessary to enable a timely transmission of other packets. A priority / importance may be associated with each haptic media packet in the queue (e.g., to enable dropping packets with the lowest priority).

[0315] An intermediate node / media sender (which may be collectively referred to as a node or intermediate node herein) may determine haptics media packet priority. One or more of the following congestion control approaches may be combined to determine the priority of a haptics media packet relative to other haptics media packets.

[0316] A MIHS layer may be used to determine that an RTP packet has a higher transmission priority than another. For example, if packet 1 has a MIHS layer value 0, it may have a higher priority than (e.g., it should be prioritized over) packet 2 with a MIHS layer value 1 .

[0317] Priority may be given to sync packets (e.g., haptics media packets that include a HTU with a sync bit set) with regard to non-sync packets. Losing a sync packet may render other haptics media packets (e.g., that may depend on the lost sync packet) useless for the application.

[0318] Priorities may be decided based on the type of MIHS unit. In examples, the importance of the HTU type may be determined by a pre-determined HTU type to importance mapping (e.g., an initialization HTU may be more important than a temporal HTU which may be more important than a spatial HTU which may be more important than a silent HTU). In other examples, the importance of the HTU type may be determined by a predetermined mapping between the tuple (HTU type, HTU syn) and an importance (e.g., (Initialization, syn) > (Temporal syn) > (Silent, syn) > (Spatial syn) > (Temporal non-syn) > (Silent, non-syn)).

[0319] When two HTUs depend on the same independent HTU, the first HTU may have a higher priority than the later HTU (e.g., because the second HTU may need the first HTU to have been decoded in order to be usable by the decoder).

[0320] A haptics media packet that has been queued for too long in the node may not be useful for the receiver (e.g., because it will arrive too late to be decoded on time to be played back to the user). Based on a configuration parameter for the flow, such a packet may be associated with the lowest priority and / or be discarded.

[0321] Transmission types may be classified into a single HTU, fragmented HTUs (FU), and / or aggregated HTUs.

[0322] For a single HTU, the priority of the haptics media packet may be obtained from characteristics of the HTU (e.g., as described in other congestion control approaches herein).

[0323] All haptics media packets containing a fragmented HTU (also referred to as “FU packets”) of a same HTU may have the same priority (e.g., relative to other haptics media packets containing other HTUs). Within the same HTU, the earlier FU packets may be prioritized over later FU packets (e.g., to preserve delivery order).

[0324] The priority of a haptics media packet containing aggregated HTUs may be based on the characteristics of the most important HTU included in the haptics media packet. The importance of HTUs within a media packet may be prioritized amongst each other, for example based on the other congestion control approaches described herein.

[0325] Feature(s) associated with network congestion control for haptic data are provided herein.

[0326] When network congestion occurs during haptic media packet transmission, network congestion control may be possible at a node, which may be an intermediate node (e.g., user plane function (UPF) or WTRU), a media sender (e.g., AS or WTRU), and / or a media receiver (e.g., WTRU). In network congestion situations, the node may determine which haptic media packets to send or drop, using haptics transport IE information.

[0327] FIG. 8 depicts an example of network congestion control.

[0328] As illustrated in FIG. 8, at 1, the media receiver, the media sender, or an operator may convey network congestion configuration to an intermediate, sender, and / or receiver node using haptics transport lEs. For example, haptics transport lEs may be configured by a network operator into an RTP middlebox. For example, haptics transport lEs may be configured by a network operator in a UPF that may use network policy information. For example, haptics transport lEs may be configured by an application provider into a media receiver / sender. The node may configure its congestion control algorithm for haptics using information from haptics transport lEs (e.g., as described herein).

[0329] At 2-4a of FIG. 8, a media receiver and a media sender may negotiate the haptics media session (e.g., as described herein). The media sender may select the haptics media stream, packetize and / or mark haptics media packets, for example based on the per-flow haptics transport lEs (e.g., the result of the negotiation). The media sender may transmit the haptics media packets towards the receiver.

[0330] At 5-6 of FIG. 8, the media sender may start streaming to the media receiver (e.g., including per- packet haptics transport lEs). Media packets may be RTP packets.

[0331] At 7 of FIG. 8, network congestion may occur in a node (e.g., in an intermediate node, in the sender, in the receiver).

[0332] At 8-10 of FIG. 8, the media sender application may send haptic media packets (A, B, C) to the node.

[0333] At 11 of FIG. 8, the node may use per-packet haptics transport lEs to run a congestion control algorithm (e.g., as described herein).

[0334] It may not be possible to transmit all packets within the required latency corresponding to the QoS flow for the haptic media flow (e.g., because of the congestion event). The transmission queue may grow large, and the node may need to drop packets which have a low priority based on policy.

[0335] The node may evaluate priority for packets in the queue (e.g., every packet in the queue) to determine priority.

[0336] The node may select which packet to send and / or in which order to send the packets in the queue based on the evaluated priority.

[0337] The node may transmit the selected packets towards the receiver and / or may drop other packets.

[0338] In this example, among the packets (A - C), the node may decide to choose what packet to transmit to the receiver based on network congestion algorithm and the intermediate node may drop other packets.

[0339] Even if some of the haptic data is missing at the media receiver (e.g., because of dropped packets), the impact on QoE may be reduced (e.g., because critical data, for example higher priority data, is delivered).

[0340] At 12 and at 13 of FIG. 8, haptic packets (A, C) may be forwarded to the media receiver, for example because they may be sync (e.g., independent) packets, while B may be a dependent packet.

[0341] Feature(s) associated with selective packet loss recovery for haptic are provided herein.

[0342] Packet loss may occur when transmitting data. An efficient method to effectively respond to packet loss may be selected based on the type and / or characteristics of data. The haptic transport lEs may be utilized to selectively request retransmission of haptic data in packet loss situations. This may enable retransmission ofpackets (e.g., a request to retransmit packets) that may have the most effect on end user QoE, while limiting network resource usage by not retransmitting other lost packets. This method may identify the importance of lost packets based on IE information. IE information may be used to request packet retransmission or delete data.

[0343] An intermediate node may detect packet loss of haptics media packet(s) using one or more of the following methods. All or some of the following methods may be combined to determine the priority of packet retransmission of a haptics media packet (e.g., relative to other haptics media packets).

[0344] FIG. 9 illustrates an example of a dependency stamp. A dependency stamp (or dependency ID) may be an indicator present in a haptics media packet. A dependency stamp may carry a value associated with a sync HTU that the haptic media packet depends on. A haptic packet transporting an independent (e.g., sync) HTU may have a dependency stamp representing the HTU it transports (e.g., a value which is unique, within a certain time range for this HTU). Haptic media packets transporting a dependent HTU may have a dependency stamp with a value representing the HTU they depend on.

[0345] The same dependency stamp information may (e.g., should) be used for all haptic media packets transporting a sync HTU and all following HTUs that may be dependent on the sync HTU for decoding. When a new sync packet is sent, the dependency stamp value for the haptics media packets transporting this HTU may be set to a new value (e.g., previous dependency stamp value plus one), and the new value may be reused for all haptics media packets until the next sync packet. The node may determine that a sync packet was lost when a node receives haptics media packets with a new dependency stamp value but no sync packet with the dependency stamp value is received.

[0346] A node (e.g., intermediary node or receiver) may use the dependency stamp to detect that a sync packet was lost. If the node was configured to apply a selective packet loss recovery algorithm, the node may request a retransmission for the lost s packet. In some cases (e.g., in high network usage situations, at all times) the node may abstain from requesting retransmission for haptics media packets which are not sync.

[0347] In some systems, the node may request retransmissions for all lost packets, and the sender may evaluate the importance of the lost packet, and only re-transmit most important packets for the end user QoE (e.g., packets carrying sync HTU or initialization HTU). For example, the sender may decide to use selective retransmission when the network usage is high, and to retransmit all lost packets when the network usage is low.

[0348] Feature(s) associated with selective packet loss recovery for haptic data are provided herein.

[0349] When packet loss occurs during haptic media packet transmission, packet loss recovery may be possible at the intermediate node (e.g., UPF or WTRU) or media receiver (e.g., WTRU), which may be referred to as the node. In packet loss situations, important packets may be selected and requested for retransmission using IE information.

[0350] FIG. 10 depicts an example of packet loss recovery. As shown in FIG. 10, the intermediate node may be distinct from the media receiver (e.g., UPF intermediate node and WTRU media receiver). In examples, the intermediate node and media receiver may be the same entity (e.g., a WTRU). If the intermediate node and the media receiver are the same entity, the interactions between the media receiver and intermediate node may be internal to the node (e.g., WTRU).

[0351] At 1 of FIG. 10, media receiver, a media sender or an operator may convey selective haptics loss recovery configuration to an intermediate or receiver node using the haptics transport lEs. For example, haptics transport lEs may be configured by a network operator into an RTP middlebox. For example, haptics transport lEs may be configured by a network operator in a UPF (e.g., using network policy information). For example, haptics transport lEs may be configured by an application provider into a media receiver. The node may configure its selective haptics loss recovery algorithm for haptics using this information (e.g., as described herein).

[0352] At 2-4a of FIG. 10 (e.g., which may be similar to and / or the same as 2-4a of FIG. 8), a media receiver and a media sender may negotiate the haptics media session. The media sender may select the haptics media stream, packetize, and / or mark haptics media packets based on the per-flow haptics transport lEs (e.g., the result of this negotiation). The media sender may transmit the haptics media packets towards the receiver.

[0353] At 5-9 of FIG. 10, the media sender may start streaming to the media receiver (e.g., including per- packet haptics transport lEs). Media packets may be RTP packets (e.g., as described herein). Some packets may be lost, for example, due to a transmission error and / or congestion (e.g., at 7 or 8 of FIG. 10).

[0354] At 10-16 of FIG. 10, the node may use per-packets haptics transport lEs to run a packet loss recovery algorithm (e.g., as described herein).

[0355] When a packet loss is detected (e.g., based on a missing RTP sequence number), the node may determine the importance of the lost packet and / or may selectively request (e.g., at 11 of FIG. 10) transmission (e.g., using an RTCP packet). The node may request transmission for important packet(s) (e.g., only important packets, such as sync packets, layer 0 packets, etc.) if network usage is high (e.g., congestion even at the node). The node may request retransmission of all packets (e.g., in other cases, such as low network usage).

[0356] When a sender receives a retransmission request, the sender may perform selective retransmission based on its configuration and / or based on current network usage (e.g., at 12 or 13 of FIG. 10). When using selective retransmission, the sender may evaluate the importance / priority of the lost packet reported by the node. The sender may retransmit important packets (e.g., sync packets and / or HTU layer 0 packets).

[0357] Upon receiving a retransmitted haptic media packet, the node may transmit it to the receiver (e.g., 14-16 of FIG. 10).

[0358] Selective loss recovery may enable a reduced impact on QoE even when a haptic media packet (e.g., sync) is missing (e.g., because the node requests retransmission to the media sender). The node and / or the sender may perform selective retransmission based on policy / configuration and / or to reduce the impact on network usage.

[0359] A sender or receiver may identify and select / mark the most important portion of the haptics traffic.

[0360] A sender (e.g., a UE or a wireless transmit receive unit (WTRU)) may perform one or more of the following.

[0361] The sender of a haptics media stream may determine different methods available to identify haptic streams and / or portions of a haptics stream that are useful to the receiver and / or may be transmitted with differentiated quality of service based on, for example, application logic and / or knowledge of receiver setups. The sender of a haptics media stream may determine different methods available to identify haptic streams and / or portions of a haptics stream that may be transmitted in different streams and / or may be grouped together in different streams. The sender may determine related parameters associated with these methods (e.g., the range of available perceptions, band frequencies in the haptics stream, a set of haptic modalities, a set of body parts targeted by the haptic stream, a set of level of details on a haptic avatar, and / or a set of target device types).

[0362] The sender may send an offer message to the receiver (e.g., including per-flow haptics transport lEs) that may include a description of the different method(s) that may be used to identify haptic streams and / or portions of a haptics stream that may be transmitted with different quality of service and / or associated parameters, and / or that may be used to identify haptics streams and / or portions of an haptics stream based on, for example their usefulness to the receiver (e.g., based on how the haptic streams match the characteristics of the user devices and / or how they contribute to the user experience).

[0363] The sender may receive an answer message from the receiver (e.g., including per-flow haptics transport lEs) that may indicate the methods and / or associated parameters that are selected by the receiver, and / or a haptic stream selected by the receiver.

[0364] The sender may select haptic stream(s) and / or mark haptics media packet(s) as per the method and associated parameters selected by the receiver. The sender may transmit the selected haptic stream(s) and / or marked haptics media packet(s), which may enable the network to apply differentiated service (e.g., for congestion control and for selective retransmission).

[0365] In examples, the sender may start new decisions based on per-flow haptic transport lEs.

[0366] The sender may utilize the device's current network status and / or flow-specific haptic transmission IE information to change priority and / or selectively transmit data. For example, if the same haptic media stream is transmitted on two frequencies, the sender may delete one frequency flow and / or use only one flow and change some parameter(s), such as layer value, depending on network conditions. The sender may utilize information coming from the device or receiver, network information, and information from the negotiation process with the receiver (e.g., for this purpose).

[0367] The sender may select, among different haptic stream(s), a haptic stream that corresponds to the method selected by the receiver (e.g., the sender may determine a specific frequency within the frequency band that the receiver may support).

[0368] The sender may process multiple haptics streams using per-flow haptics transport lEs. Decisions (e.g., all decisions) may be subject to change through renegotiation depending on the situation.

[0369] A receiver (e.g., a haptic rendering device) may perform one or more of the following.

[0370] The receiver of a haptics media stream may receive an offer message from the sender (e.g., including per-flow Haptics Transport lEs) that may include a description of the different methods that may be used to identify haptics streams and / or portions of a haptics stream that may be transmitted with different quality of service, and associated parameters, and / or that may be used to identify haptics streams and / or portions of an haptics stream based on their usefulness to the receiver (e.g., based on how the haptics streams match the characteristics of the user devices and / or how the haptics streams contribute to the user experience).

[0371] The receiver may select suitable method(s) and associated parameters to identify haptics streams and / or portions of the haptics stream that may be transmitted with different quality of service among the methods and associated parameters from the sender (e.g., based on application logic and / or knowledge oflocal haptics devices). The receiver may select, using the method(s) and associated parameters, one or more haptic streams from the offer message, that are suitable for streaming to the receiver.

[0372] The receiver may send an answer message to the sender, including per-flow haptics transport lEs that indicate the selected method(s) and related parameters, and / or selected haptic streams.

[0373] The receiver may receive (e.g., from the sender) the haptics media stream (e.g., including the selected haptics streams) with media packets marked as per the method selected by the receiver, which may enable the network to apply differentiated service (e.g., for congestion control and for selective retransmission).

[0374] In examples, the receiver may process multiple haptic streams using per-flow haptics transport lEs.

[0375] The receiver may notify the sender of a change in status if the receiver devices’ specifications or environment changes.

[0376] The receiver may request to change the data type from the sender.

[0377] Feature(s) associated with network congestion control for haptic data are provided herein.

[0378] An intermediate node or a sender (e.g., WTRU, UPF) may perform one or more of the following.

[0379] The node may receive a configuration for a haptics stream.

[0380] The node may receive haptics media packets from the media sender application (e.g., application server), that may include per-packet haptics transport lEs. Haptics media packets may be RTP packets. Per- packet haptics transport IE may be a sync indication, haptics transport unit type, dependency stamp, haptics transport unit layer, haptics sequence numbers, or other haptics transport indication or lEs (e.g., as described herein).

[0381] The node may experience a congestion event, which may reduce the node’s transmission capacity (e.g., temporarily).

[0382] The node may use per-packets haptics transport lEs to run a congestion control algorithm for haptic data, which may include determining the priority of haptics media packets relative to each other and / or deciding whether to transmit and / or drop haptics media packets.

[0383] The node may transmit haptics media packets to the media receiver (e.g., WTRU), where the transmitted haptics media packets may enable the receiver / UE to play the haptics experience, and where the dropped haptics media packets may have reduced influence on QoE.

[0384] In examples, configuration may include per-flow haptics transport lEs. And an intermediate node may use this configuration to run a congestion control algorithm for haptic data.

[0385] Using the haptics transport lEs, the media receiver or media sender may convey the configuration to the intermediate node. The intermediate node may configure its congestion control algorithm for haptics using this information.

[0386] Feature(s) associated with packet loss recovery for haptics media streams are provided herein.

[0387] An intermediate node or receiver (e.g., WTRU, UPF) may perform one or more of the following.

[0388] The node may receive a configuration for a haptics stream.

[0389] The node may receive haptics media packets from the media sender (e.g., application server), which may include per-packet haptics transport lEs. Haptics media packets may be RTP packets.

[0390] The node may use per-packets haptics transport lEs and packet loss recovery configuration lEs to run a packet loss recovery algorithm, which may include determining selective packet retransmission requests.

[0391] If the node detects haptics media packet loss and determines this media packet should be retransmitted, the node may send a retransmission request message to the media sender.

[0392] Upon reception of a retransmitted haptic media packet, the node may transmit the haptic media packet to the media receiver (e.g., WTRU). Due to the selective retransmission of the most QoE-important haptics media packets, the impact on the haptic application's QoE may be kept low (e.g., while limiting the number of retransmissions).

[0393] In examples, the configuration may include per-flow haptics transport lEs. The node may use this configuration to run a packet loss recovery algorithm.

[0394] Using the haptics transport lEs, the media receiver or media sender may convey configuration to the node (e.g., in some cases the node may be a trusted intermediary in an SDP exchange). The node may configure its packet loss recovery algorithm for haptics using this information.

[0395] Feature(s) associated with haptic silence suppression are provided herein.

[0396] A receiver (e.g., a UE or a WTRU) may perform one or more of the following.

[0397] The node may receive an SDP answer message, which may include an indication to use haptic silence suppression. The indication to use haptic silence suppression may include a number N of initial silent MIHS units (e.g., where N is a positive integer).

[0398] The node may receive media packet(s) including N silent MIHS units at the same MIHS layer and may keep the media packets timing and MIHS layer in its internal state.

[0399] The node may determine that one or more media packets were not received, where the missing media packets follow the N silent MIHS units on the same MIHS layer (e.g., based on a timeout).

[0400] The node may synthesize a silent MIHS unit, contiguously following the N silent MIHS units in time and at the same MIHS layer.

[0401] The node may provide the synthesized silent MIHS unit to the decoder.

[0402] The definition of the RTP header may follow a header format represented in Table 5. The three header values may be overridden.Table 5: an example RTP header for haptic

[0403] Payload type (PT) may be 7 bits. Types may be specified to indicate that the payload contains haptic data (e.g., in addition to the existing types). For example, a type called 'Haptic' may be defined and may use the dynamic payload type field.

[0404] Time Stamp (TS) may be 32 bits. Timestamps may be calculated with packet durations. The timestamp may allow finding the exact location of the packet.

[0405] Marker bit (M) may be 1 bit. This bit may be used to mark the first non-silent haptic media packet (e.g., after period of silence). This may enable jitter buffer adaptation and haptics device washout (e.g., reset to a neutral position) prior to the beginning of the burst with minimal impact on the quality of experience for the end user.

[0406] Exemplary RTP payload Header format for haptic may be provided.

[0407] The structure below is the RTP Payload Header for Haptic. Table 6 describes an example RTP payload header.Table 6: RTP payload header for haptic.

[0408] Sync type may be represented as D and may be 1 bit. The sync type may be used to indicate whether a MIHS unit included in the haptics media packet payload (e.g., RTP payload) is a sync packet or not. A sync packet of haptic data may be a packet containing sync information for providing a haptic effect and may be used until sync information is changed. Sync information may be important information and may refer to non-sync packets. The sync type may be used to prioritize haptics media packets.

[0409] Unit type may be represented as UT and may be 3 bits. The Unit type may indicate a MIHS unit type. Based on the unit type, a user may define processing methods based on packet importance. Through the unit type, it may be possible to check what type of MIHS unit is included. The importance of MIHS unit may be analyzed based on unit type to enable effective haptic media packet transmission. Unit type may be represented as shown in Table 7.

[0410] MIHS layer may be represented as L and may be 4 bits. MIHS layer may be used to indicate the priority order of MIHS unit as determined by the haptic sender (e.g., haptics codec), for example, based on application-specific needs. The lower the MIHS layer, the higher the priority may be given to packet processing (e.g., by intermediate nodes and / or by receivers). For example, MIHS layer value may be used to determine the importance of a haptics media packet when network congestion occurs. In examples, when haptics media packets are transmitted along with other types of media packets, such as audio / video packets, that are also associated with MIHS layer, the haptics media packets MIHS layer may be used with the audio / video layer to determine processing priority between haptics / audio / video media packets. MIHS layer values may be integers, with, for example, 0 as the highest priority.

[0411] Exemplary Payload Structures

[0412] Two different types of RTP packet payload structures may be specified. The receiver may identify the payload structure using the unit type field. The first structure may be for a single MIHS unit and the other may be for fragments of a MIHS unit.

[0413] Table 8 represents an example unit type. The unit type field may represent both the payload structure and, in the single MIHS unit case, the type of MIHS unit present in the payload.Table 8: example unit type fieldUnit Payload NameType Structure0 N / A Reserved1 Single Initialization MIHS Unit2 Single Temporal MIHS Unit3 Single Spatial MIHS Unit4 Single Silent MIHS Unit7 Frag Fragmented Packet

[0414] Table 9 represents example payload structures.Table 9: example RTP payload structures for haptic+-+| RTP Header |+-+ +-+| RTP Header | | RTP payload Header | + -+I (UT = Frag) || RTP payload Header |+-++-+| FU Header || RTP payload |+-+| (Single MIHS unit) | | RTP Payload | + -++-+(a) Single MIHS RTP (b) Fragmented RTP

[0415] Table 10 represents an exemplary single MIHS unit payload structure. The RTP packet may contain an RTP payload header followed by one single MIHS unit. The RTP payload may contain a MIHS unit.Table 10: example single MIHS unit structure

[0416] Table 11 represents an exemplary fragmented MIHS unit payload structure. The RTP packet may contain a RTP payload header followed by a fragmented unit (FU) header, followed by a MIHS unit fragment. The value of the UT field of the RTP payload header may be 7.Table 11 : example fragmentation MIHS unit structure

[0417] FUs may be used to enable fragmenting a single MIHS unit into multiple RTP packets. Fragments of the same MIHS unit may be sent in consecutive order with ascending RTP sequence numbers (e.g., with no other RTP packets within the same RTP stream being sent between the first and last fragment). FUs may not be nested (e.g., an FU may not contain a subset of another FU).

[0418] Table 12 represents an FU header.Table 12: example fragmentation unit header

[0419] The fragmented unit start may be represented as FUS and may be 1 bit. The fragmented unit start may be set to 1 for the first fragment and 0 for the other fragments.

[0420] The fragmented unit end may be represented as FUE and may be 1 bit. The fragmented unit end may be set to 1 for the last fragment, and 0 for the other fragments.

[0421] Reserved bits may be represented as RSV and may be 3 bits. Reserved bits may be set to 0 by the sender and ignored by the receiver.

[0422] The unit type field may be represented as UT and may be 3 bits. The unit type field may indicate the type of the MIHS unit this fragment belongs to and may use values using values defined in Table 8.

[0423] Feature(s) associated with end point behavior for haptic are provided herein.

[0424] For haptic frames a media sender may set durations with the same value for all non-zero duration MIHS units between initialization MIHS units (e.g., to make the decoder more robust to RTP packet loss).

[0425] For haptic silence suppression silent MIHS units may represent a period of haptic silence. A sender may send the first one or more silent units at the beginning of a haptic silence (e.g., because this may help the receiver interpret the lack of received traffic as silence with high probability). A sender may not send subsequent consecutive silent units (e.g., to save network resources).

[0426] For parameters, a type name may be haptics and a subtype name may be hmpg.

[0427] Payload format parameters may include one or more of the following: hmpg-version (e.g., may provide the year of the edition and amendment of the specifications followed by this RTP payload type); hmpg- profile (e.g., name of the profile used to generate the encoded stream); hmpg-level (e.g., number of the level used to generate the encoded stream); hmpg-date (e.g., indicates the creation date of this haptic experience); hmpg-description (e.g., the user-defined description of this haptic experience); hmpg-timescale (e.g., number of ticks per second); hmpg-avatar-lods (e.g., indicates which levels of details to use for the avatar(s)); hmpg- avatar-types (e.g., indicates the type of haptic perception represented by the avatar(s)); hmpg-perception- modalities (e.g., provides the types of haptic perception (e.g., pressure, acceleration, velocity, position, temperature, etc.)); hmpg-dvc-body-part-mask (e.g., binary mask specifying the location of the devices or actuators on the body); hmpg-dvc-maximum-frequency (e.g., maximum frequency of the actuators (Hz); hmpg- dvc-minimum-frequency (e.g., minimum frequency of the actuator (Hz); hmpg-dvc-resonance-frequency (e.g., resonance frequency of the actuator (Hz); hmpg-dvc-max-amplitude (e.g., maximum amplitude value of the targeted device); hmpg-dvc-impedance number (e.g., impedance of the actuator (Q)); hmpg-dvc-max-voltage (e.g., maximum voltage of the actuator (V)); hmpg-dvc-max-current (e.g., maximum current of the actuator (A)); hmpg-dvc-max-displacement (e.g., maximum displacement of the actuator (mm)); hmpg-dvc-weight (e.g., weight of the device (Kg)); hmpg-dvc-size (e.g., size of the device (mm)); hmpg-dvc-custom (e.g., user-defined data); hmpg-dvc-type (e.g., indicates the type of actuator); or hmpg-silence-suppression (e.g., indicates whether silence suppression should be used).

[0428] The above defined payload format media types may be mapped to the corresponding fields in the SDP. The media name in the "m=" line of SDP may be haptics. The encoding name in the "a=rtpmap" line of SDP may be hmpg. The clock rate in the "a=rtpmap" line may be any sampling rate (e.g., 8000).

[0429] Some parameters may be included in the "a=fmtp" line of SDP. This may be expressed as a media type string (e.g., in the form of a semicolon-separated list of parameter=value pairs).

[0430] An example of media representation corresponding to a hmpg RTP payload in SDP may be as follows:m=haptics 43291 UDP / TLS / RTP / SAVPF 115 a=rtpmap:115 hmpg / 8000 a=fmtp:115 hmpg-profile-id=1;hmpg-level-id=1;hmpg-version=2023

[0431] Feature(s) associated with SDP offer / answer considerations are provided herein.

[0432] When using the offer / answer procedure to negotiate the use of haptic, one or more of the following considerations may apply.

[0433] The haptic signal may be sampled at different rates. A sample rate may be 8000Hz.

[0434] The parameter hmpg-ver may indicate the version of the haptic standard specification. The parameter hmpg-profile may be used to restrict the number of tools used (e.g., the simple-parametric profile fits enable simpler implementations than the main profile). If it is not specified, the most general profile “main” may be assumed, although the sender and receiver may use a specific value based on an out-of-band agreement. The parameter hmpg-lvl may be used to further characterize implementations within a given profile (e.g., according to the maximum supported number of channels, bands, and perceptions). If it is not specified, the most general level “2” may be assumed, although the sender and receiver may use a specific version based on an out-of-band agreement.

[0435] Other parameters may be used to indicate bitstream properties and / or receiver capabilities. The parameters hmpg-maxlod, hmpg-avtypes, hmpg-bodypartmask, hmpg-maxfreq, hmpg-minfreq, hmpg- dvctypes, and hmpg-modalities may be sent by a sender to reflect the characteristics of bitstreams and may be set by a receiver to reflect the nature and capabilities of local actuator devices, or a preferred set of bitstream properties. For example, different receivers may have different sets of local actuators, in which case these parameters may be used to select a stream adapted to the receiver. Some receivers may indicate a preference for a set of bitstream properties such as perceptions, min / max frequency, and / or body-part-mask, which may contribute the most to the user experience for a given application, in which case these parameters may be used to select a stream which may include and / or prioritize those properties. Per-flow haptic transport lEs (e.g., as defined herein) may be used as a parameter to indicate bitstream properties and / or receiver capabilities.

[0436] The parameter hmpg-silencesupp may be used to indicate sender and / or receiver capabilities or preferences. This parameter may indicate whether silence suppression should be used.

[0437] Feature(s) associated with declarative SDP considerations are provided herein.

[0438] When haptic content over RTP is offered with SDP in a declarative style, the parameters capable of indicating bitstream properties and receiver capabilities may be used to indicate (e.g., only) bitstreamproperties. For example, in this case, the parameters hmpg-maxlod, hmpg-bodypartmask, hmpg-maxfreq, hmpg-minfreq, hmpg-dvctypes, and hmpg-modalities (and / or other parameters defined herein) may declare the values used by the bitstream (e.g., not the capabilities for receiving bitstreams). A receiver of the SDP may be required to support all parameters and values of the parameters provided; otherwise, the receiver may reject or not participate in the session. The creator of the session may use values that are expected to be supported by the receiving application.

[0439] Feature(s) associated with congestion control considerations are provided herein.

[0440] The general congestion control considerations for transporting RTP data may apply to HMPG haptics over RTP.

[0441] Network bandwidth may be adapted by adjusting the encoder bit rate and / or by adjusting the stream content (e.g., level of detail, body parts, actuator frequency range, target device types, modalities). MIHS units may be allocated to different layers based on their content using the layer field of the RTP payload header (e.g., to prioritize haptic data contributing the most to the user experience).

[0442] Although features and elements are described above in particular combinations, one of ordinary skill in the art will appreciate that each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable media include electronic signals (transmitted over wired or wireless connections) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

CLAIMS1 . A haptic sending device comprising: a processor configured to: send an offer message to a haptic receiving device, wherein the offer message comprises a set of parameters associated with each portion of a plurality of portions of a haptic media bitstream; receive an answer message from the haptic receiving device indicating a selection of a subset of parameters from the set of parameters associated with each portion of the plurality of portions of the haptic media bitstream; based on the selection of the subset of parameters, select a portion of the haptic media bitstream; and send the selected portion of the haptic media bitstream to the haptic receiving device.

2. The haptic sending device of claim 1 , wherein the processor is configured to determine the set of parameters based on at least one of an application logic or information about the haptic receiving device.

3. The haptic sending device of claim 1 or claim 2, wherein the set of parameters indicates at least one of a range of available band frequencies, a level of details to be used for avatars, a set of haptic perceptions represented by one or more avatars, a location of one or more devices or actuators on a body, a maximum frequency value of an actuator, a minimum frequency value of the actuator, an actuator type, or a set of haptic perception modalities.

4. The haptic sending device of any one of claims 1-3, wherein the offer message or the answer message is communicated via a per-packet haptics transport information element (IE).

5. The haptic sending device of claim 4, wherein the per-packet haptics transport IE is at least one of a sync type IE, a unit type IE, a dependency stamp IE, a dependency identifier (ID) IE, a haptic transport unit (HTU) layer IE, a transmission type IE, or a haptic sequence number IE.

6. The haptic sending device of any one of claims 4 or 5, wherein the per-packet haptics transport IE comprises an indication of at least one of: a sync type indicating whether a MPEG-I Haptic Streaming (MIHS) unit is a sync packet;a unit type indicating a priority for the per-packet haptics transport IE; or a MIHS layer indicating a priority order of the MIHS unit.

7. The haptic sending device of claim 5, wherein the transmission type IE indicates transmission type of a haptic media packet, and wherein the transmission type is of a single MPEG-I Haptic Streaming (MIHS) packet mode or a fragmented MIHS packet mode.

8. The haptic sending device of claim 5, wherein the processor is configured to determine a priority value associated with an HTU, and wherein the HTU layer IE indicates a priority level associated with the HTU.

9. The haptic sending device of any one of claims 1 -8, wherein the offer message or the answer message comprises a per-flow haptics transport IE.

10. The haptic sending device of claim 9, wherein the per-flow haptics transport IE comprises at least one of a frequency range IE, a perception modality IE, a type of actuator IE, a layer information IE, a haptics-encoding- method IE, a body part mask IE, a haptic silence suppression IE, a default haptics signal IE, a version IE, a profile IE, a level IE, a layer vs. sync priority IE, a media-type-priority IE, a haptics congestion control configuration IE, or a selective haptics loss recovery configuration IE.11 . The haptic sending device of any of claims 1-6 or claim 8 or claim 9, wherein the haptic media bitstream comprises a haptic media packet, wherein the haptic media packet is marked with one parameter of the parameter set, and wherein the parameter enables the haptic receiving device or an intermediate device to apply a differentiated service.

12. The haptic sending device of any one of claims 1-11 , wherein each portion of the plurality of the portions of the haptic media bitstream is associated with a differentiated quality of service (QoS).

13. A haptic receiving device comprising: a processor configured to: receive an offer message from a haptic sending device, wherein the offer message comprises a set of parameters associated with each portion of a plurality of portions of a haptic media bitstream; select a subset of parameters from the received set of parameters;send an answer message to the haptic sending device indicating a selection of the subset of parameters from the set of parameters; and receive a portion of the haptic media bitstream from the haptic sending device, wherein the portion of the haptic media bitstream is associated with the subset of the parameters.

14. The haptic receiving device of claim 13, wherein the processor being configured to select the subset of parameters from the received set of parameters comprises the processor being configured to select the subset of parameters from the received set of parameters based on application logic or information associated with at least one local device of the haptic receiving device.

15. The haptic receiving device of claim 13 or claim 14, wherein the set of parameters indicates at least one of a range of available band frequencies, a level of details to be used for avatars, a set of haptic perceptions represented by one or more avatars, a location of one or more devices or actuators on a body, a maximum frequency value of an actuator, a minimum frequency value of the actuator, an actuator type, or a set of haptic perception modalities.

16. The haptic receiving device of any one of claims 13-15, wherein the offer message or the answer message is communicated via a per-packet haptics transport information element (IE).

17. The haptic receiving device of claim 16, wherein the per-packet haptics transport IE is at least one of a sync type IE, a unit type IE, a dependency stamp IE, a dependency identifier (ID) IE, a haptic transport unit (HTU) layer IE, a transmission type IE, or a haptic sequence number IE.

18. The haptic receiving device of any one of claims 16 or 17, wherein the per-packet haptics transport IE comprises an indication of at least one of: a sync type indicating whether a MPEG-I Haptic Streaming (MIHS) unit is a sync packet; a unit type indicating a priority for the per-packet haptics transport IE; or a MIHS layer indicating a priority order of the MIHS unit.

19. The haptic receiving device of claim 17, wherein the transmission type IE indicates transmission type of a haptic media packet, and wherein the transmission type is of a single MPEG-I Haptic Streaming (MIHS) packet mode or a fragmented MIHS packet mode.

20. The haptic receiving device of claim 17, wherein the HTU layer IE indicates a priority value associated with an HTU.

21. The haptic receiving device of any one of claims 13-20, wherein the offer message or the answer message comprises a per-flow haptics transport IE.

22. The haptic receiving device of claim 21 , wherein the per-flow haptics transport IE comprises at least one of a frequency range IE, a perception modality IE, a type of actuator IE, a layer information IE, a haptics-encoding- method IE, a body part mask IE, a haptic silence suppression IE, a default haptics signal IE, a version IE, a profile IE, a level IE, a layer vs. sync priority IE, a media-type-priority IE, a haptics congestion control configuration IE, or a selective haptics loss recovery configuration IE.

23. The haptic receiving device of any of claims 13-18 or claim 20 or claim 21, wherein the haptic media bitstream comprises a haptic media packet, wherein the haptic media packet is marked with one parameter of the parameter set, and wherein the parameter enables the haptic receiving device or an intermediate device to apply a differentiated service.

24. The haptic receiving device of any one of claims 13-23, wherein each portion of the plurality of the portions of the haptic media bitstream is associated with a differentiated quality of service (QoS).

25. A method implemented by a haptic sending device comprising: sending an offer message to a haptic receiving device, wherein the offer message comprises a set of parameters associated with each portion of a plurality of portions of a haptic media bitstream; receiving an answer message from the haptic receiving device indicating a selection of a subset of parameters from the set of parameters associated with each portion of the plurality of portions of the haptic media bitstream; based on the selection of the subset of parameters, selecting a portion of the haptic media bitstream; and sending the selected portion of the haptic media bitstream to the haptic receiving device.

26. The method of claim 25, further comprising determining the set of parameters based on at least one of an application logic or information about the haptic receiving device.

27. The method of claim 25 or claim 26, wherein the set of parameters indicates at least one of a range of available band frequencies, a level of details to be used for avatars, a set of haptic perceptions represented by one or more avatars, a location of one or more devices or actuators on a body, a maximum frequency value of an actuator, a minimum frequency value of the actuator, an actuator type, or a set of haptic perception modalities.

28. The method of any one of claims 25-27, wherein the offer message or the answer message is communicated via a per-packet haptics transport information element (IE).

29. The method of claim 28, wherein the per-packet haptics transport IE is at least one of a sync type IE, a unit type IE, a dependency stamp IE, a dependency identifier (ID) IE, a haptic transport unit (HTU) layer IE, a transmission type IE, or a haptic sequence number IE.

30. The method of any one of claims 28 or 29, wherein the per-packet haptics transport IE comprises an indication of at least one of: a sync type indicating whether a MPEG-I Haptic Streaming (MIHS) unit is a sync packet; a unit type indicating a priority for the per-packet haptics transport IE; or a MIHS layer indicating a priority order of the MIHS unit.31 . The method of claim 29, wherein the transmission type IE indicates transmission type of a haptic media packet, and wherein the transmission type is of a single MPEG-I Haptic Streaming (MIHS) packet mode or a fragmented MIHS packet mode.

32. The method of claim 29, further comprising determining a priority value associated with an HTU, and wherein the HTU layer IE indicates a priority level associated with the HTU.

33. The method of any one of claims 25-32, wherein the offer message or the answer message comprises a per-flow haptics transport IE.

34. The method of claim 33, wherein the per-flow haptics transport IE comprises at least one of a frequency range IE, a perception modality IE, a type of actuator IE, a layer information IE, a haptics-encoding-method IE, a body part mask IE, a haptic silence suppression IE, a default haptics signal IE, a version IE, a profile IE, a level IE, a layer vs. sync priority IE, a media-type-priority IE, a haptics congestion control configuration IE, or aselective haptics loss recovery configuration IE.

35. The method of any of claims 1 -6 or claim 32 or claim 33, wherein the haptic media bitstream comprises a haptic media packet, wherein the haptic media packet is marked with one parameter of the set of parameters, and wherein the parameter enables the haptic receiving device or an intermediate device to apply a differentiated service.

36. The method of any one of claims 25-35, wherein each portion of the plurality of the portions of the haptic media bitstream is associated with a differentiated quality of service (QoS).

Images