Supporting actuator-specific haptic sensation synthesis in split rendering for extended reality media data

Abstract

An example device for participating in an augmented reality (AR) media session is associated with an actuator device. The device for participating in the AR media session may be, for example, a client device (e.g., a user equipment (UE) device) or a split rendering server device. The device participating in the AR media session may include a memory configured to store media data and a processing system implemented in circuitry and configured to: retrieve data from an actuator device database representative of one or more haptic representations supported by the actuator device; and send haptic information as part of the AR media session, the haptic information conforming to one of the one or more haptic representations supported by the actuator device. In this manner, the device may participate in AR media sessions using, and communicate with actuator devices that implement, a variety of different haptic representations.

Description

Qualcomm Ref. No. 2408048WO 1SUPPORTING ACTUATOR-SPECIFIC HAPTIC SENSATION SYNTHESISIN SPLIT RENDERING FOR EXTENDED REALITY MEDIA DATA

[0001] This application claims priority to India Application No. 202441075391, filed October 4, 2024, the entire contents of which are hereby incorporated by reference.TECHNICAL FIELD

[0002] This disclosure relates to transport of media data, and more particularly, to split rendering of extended reality media data.BACKGROUND

[0003] Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite radio telephones, video teleconferencing devices, and the like. Digital video devices implement video compression techniques, such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263 or ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265 (also referred to as High Efficiency Video Coding (HEVC)), and extensions of such standards, to transmit and receive digital video information more efficiently.

[0004] After media data has been encoded, the media data may be packetized for transmission or storage. The video data may be assembled into a media file conforming to any of a variety of standards, such as the International Organization for Standardization (ISO) base media file format and extensions thereof.SUMMARY

[0005] In general, this disclosure describes techniques related to performing split rendering of extended reality (XR) media data. XR media data may include any or all of augmented reality (AR), virtual reality (VR), or mixed reality (MR) media data. A media communication session may include XR media data, as well as other types of media data, such as images, video, sound, or the like. A user may wear a head mounted display (HMD) including left and right eye displays, and may include sensors (e.g.,1616-554WO01Qualcomm Ref. No. 2408048WO 2 cameras, infrared sensors, or the like) that may be used to determine orientation of a user’s view direction. The view direction in real world coordinates may be used to render appropriate views of the virtual scene, such that real world movements may be translated into corresponding camera movements relative to the virtual scene. A user may navigate the virtual scene through head and body movements, and / or interact with the virtual scene using controller devices including buttons joysticks, trackpads, sensors for detecting real world movement, gyroscopes, or the like. Furthermore, the controllers and / or HMD may include actuators for delivering haptic feedback to the user. This disclosure describes techniques for supporting actuator-specific haptic sensation synthesis during split rendering of an XR media session.

[0006] In one example, a method of communicating media data includes: retrieving, by a device participating in an augmented reality (AR) media session, data from an actuator device database representative of one or more haptic representations supported by an actuator device associated with the device participating in the AR media session; and sending, by the device participating in the AR media session, haptic information as part of the AR media session, the haptic information conforming to one of the one or more haptic representations supported by the actuator device.

[0007] In another example, a device for participating in an augmented reality (AR) media session, where the device is associated with an actuator device, includes: a memory configured to store media data; and a processing system implemented in circuitry and configured to: retrieve data from an actuator device database representative of one or more haptic representations supported by the actuator device; and send haptic information as part of the AR media session, the haptic information conforming to one of the one or more haptic representations supported by the actuator device.

[0008] In another example, a device participating in an augmented reality (AR) media session, which is associated with an actuator device, includes: means for retrieving data from an actuator device database representative of one or more haptic representations supported by an actuator device associated with the device participating in the AR media session; and means for sending haptic information as part of the AR media session, the haptic information conforming to one of the one or more haptic representations supported by the actuator device.

[0009] In another example, a computer-readable storage medium has stored thereon instructions that, when executed by a processor of a device for participating in an augmented reality (AR) media session, where the device is associated with an actuator1616-554WO01Qualcomm Ref. No. 2408048WO 3 device, cause the processor to: retrieve data from an actuator device database representative of one or more haptic representations supported by the actuator device; and send haptic information as part of the AR media session, the haptic information conforming to one of the one or more haptic representations supported by the actuator device.

[0010] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF DRAWINGS

[0011] FIG. 1 is a block diagram illustrating an example network including various devices for performing the techniques of this disclosure.

[0012] FIG. 2 is a block diagram illustrating an example computing system that may perform split rendering techniques of this disclosure.

[0013] FIG. 3 is a flowchart illustrating an example method of performing split rendering according to techniques of this disclosure.

[0014] FIG. 4 is a conceptual diagram illustrating a system for transmitting XR media data including haptic representations.

[0015] FIG. 5 is a flow diagram illustrating an example actuator registration process according to techniques of this disclosure.

[0016] FIG. 6 is a flow diagram illustrating an example technique for retrieving actuator information to generate actuator-type-specific haptic representations according to techniques of this disclosure.

[0017] FIG. 7 is a flow diagram illustrating an example technique for generating actuator-type-specific haptic representations per techniques of this disclosure.DETAILED DESCRIPTION

[0018] In general, this disclosure describes techniques for performing split rendering of augmented reality (AR) media data or other extended reality (XR) media data, such as mixed reality (MR) or virtual reality (VR). A split rendering server may perform at least part of a rendering process to form rendered images, then stream the rendered images to a display device, such as AR glasses or a head mounted display (HMD). In general, a user may wear the display device, and the display device may capture pose information,1616-554WO01Qualcomm Ref. No. 2408048WO 4 such as a user position and orientation / rotation in real world space, which may be translated to render images for a viewport in a virtual world space.

[0019] Split rendering may enhance a user experience through providing access to advanced and sophisticated rendering that otherwise may not be possible or may place excess power and / or processing demands on AR glasses or a user equipment (UE) device. In split rendering all or parts of the 3D scene are rendered remotely on an edge application server, also referred to as a “split rendering server” in this disclosure. The results of the split rendering process are streamed down to the UE or AR glasses for display. The spectrum of split rendering operations may be wide, ranging from full prerendering on the edge to offloading partial, processing-extensive rendering operations to the edge.

[0020] The display device (e.g., UE / AR glasses) may stream pose predictions to the split rendering server at the edge. The display device may then receive rendered media for display from the split rendering server. The XR runtime may be configured to receive rendered data together with associated pose information (e.g., information indicating the predicted pose for which the rendered data was rendered) for proper composition and display. For instance, the XR runtime may need to perform pose correction to modify the rendered data according to an actual pose of the user at the display time. This disclosure describes techniques for conveying render pose information together with rendered images, e.g., in the form of a Real-time Transport Protocol (RTP) header extension. In this manner, the display device can accurately correct and display rendered images when the images were rendered by a separate device, e.g., for split rendering. This may allow advanced rendering techniques to be performed by the split rendering server while also presenting images that accurately reflect a user pose (e.g., position and orientation / rotation) to the user.

[0021] Furthermore, in some cases, an XR media session may include haptic feedback. For example, controllers may vibrate to confirm inputs, when a user’s hand touches a virtual object, or in other such cases to provide physical feedback to the user. Various developers and manufacturers of controllers or other physical devices used for XR media sessions have developed proprietary haptic representations. While it may be unrealistic to expect that such manufacturers should implement haptic standards (e.g., per MPEG haptic standards), the techniques of this disclosure may be used to enable support for various types of actuators without changing the actuators in a unified way. In this manner, devices conforming to manufacturer proprietary haptic representations1616-554WO01Qualcomm Ref. No. 2408048WO 5 may nevertheless be used during an XR media session that uses standardized haptic representations.

[0022] FIG. 1 is a block diagram illustrating an example network 10 including various devices for performing the techniques of this disclosure. In this example, network 10 includes user equipment (UE) devices 12, 14, call session control function (CSCF) 16, multimedia application server (MAS) 18, data channel signaling function (DCSF) 20, multimedia resource function (MRF) 26, and augmented reality application server (AR AS) 22. MAS 18 may correspond to a multimedia telephony application server, an IP Multimedia Subsystem (IMS) application server, or the like.

[0023] UEs 12, 14 represent examples of UEs that may participate in an AR communication session 28 (which may also be referred to as an “XR communication session”). AR communication session 28 may generally represent a communication session during which users of UEs 12, 14 exchange voice, video, and / or AR data (and / or other XR data). For example, AR communication session 28 may represent a conference call during which the users of UEs 12, 14 may be virtually present in a virtual conference room, which may include a virtual table, virtual chairs, a virtual screen or white board, or other such virtual objects. The users may be represented by avatars, which may be realistic or cartoonish depictions of the users in the virtual AR scene. The users may interact with virtual objects, which may cause the virtual objects to move or trigger other behaviors in the virtual scene. Furthermore, the users may navigate through the virtual scene, and a user’s corresponding avatar may move according to the user’s movements or movement inputs. In some examples, the users’ avatars may include faces that are animated according to the facial movements of the users (e.g., to represent speech or emotions, e.g., smiling, thinking, frowning, or the like).

[0024] UEs 12, 14 may exchange AR media data related to a virtual scene, represented by a scene description. Users of UEs 12, 14 may view the virtual scene including virtual objects, as well as user AR / XR data, such as avatars, shadows cast by the avatars, user virtual objects, user provided documents such as slides, images, videos, or the like, or other such data. Ultimately, users of UEs 12, 14 may experience an AR call from the perspective of their corresponding avatars (in first or third person) of virtual objects and avatars in the scene.

[0025] UEs 12, 14 may collect pose data for users of UEs 12, 14, respectively. For example, UEs 12, 14 may collect pose data including a position of the users,1616-554WO01Qualcomm Ref. No. 2408048WO 6 corresponding to positions within the virtual scene, as well as an orientation of a viewport, such as a direction in which the users are looking (i.e., an orientation of UEs 12, 14 in the real world, corresponding to virtual camera orientations). UEs 12, 14 may provide this pose data to AR AS 22 and / or to each other.

[0026] CSCF 16 may be a proxy CSCF (P-CSCF), an interrogating CSCF (I-CSCF), or serving CSCF (S-CSCF). CSCF 16 may generally authenticate users of UEs 12 and / or 14, inspect signaling for proper use, provide quality of service (QoS), provide policy enforcement, participate in session initiation protocol (SIP) communications, provide session control, direct messages to appropriate application server(s), provide routing services, or the like. CSCF 16 may represent one or more I / S / P CSCFs.

[0027] MAS 18 represents an application server for providing voice, video, and other telephony services over a network, such as a 5G network. MAS 18 may provide telephony applications and multimedia functions to UEs 12, 14.

[0028] DCSF 20 may act as an interface between MAS 18 and MRF 26, to request data channel resources from MRF 26 and to confirm that data channel resources have been allocated. DCSF 20 may receive event reports from MAS 18 and determine whether an AR communication service is permitted to be present during a communication session (e.g., an IMS communication session).

[0029] MRF 26 may be an enhanced MRF (eMRF) in some examples. In general, MRF 26 generates scene descriptions for each participant in an AR communication session. MRF 26 may support an AR conversational service, e.g., including providing transcoding for terminals with limited capabilities. MRF 26 may collect spatial and media descriptions from UEs 12, 14 and create scene descriptions for symmetrical AR call experiences. In some examples, rendering unit 24 may be included in MRF 26 instead of AR AS 22, such that MRF 26 may provide remote AR rendering services, as discussed in greater detail below.

[0030] MRF 26 may request data from UEs 12, 14 to create a symmetric experience for users of UEs 12, 14. The requested data may include, for example, a spatial description of a space around UEs 12, 14; media properties representing AR media that each of UEs 12, 14 will be sending to be incorporated into the scene; receiving media capabilities of UEs 12, 14 (e.g., decoding and rendering / hardware capabilities, such as a display resolution); and information based on detecting location, orientation, and capabilities of physical world devices that may be used in an audio-visual communication sessions. Based on this data, MRF 26 may create a scene that defines placement of each user and1616-554WO01Qualcomm Ref. No. 2408048WO 7AR media in the scene (e.g., position, size, depth from the user, anchor type, and recommended resolution / quality); and specific rendering properties for AR media data (e.g., if 2D media should be rendered with a “billboarding” effect such that the 2D media is always facing the user). MRF 26 may send the scene data to each of UEs 12, 14 using a supported scene description format.

[0031] ARAS 22 may participate in AR communication session 28. For example, AR AS 22 may provide AR service control related to AR communication session 28. AR service control may include AR session media control and AR media capability negotiation between UEs 12, 14 and rendering unit 24.

[0032] AR AS 22 also includes rendering unit 24, in this example. Rendering unit 24 may perform split rendering on behalf of at least one of UEs 12, 14. In some examples, two different rendering units may be provided. In general, rendering unit 24 may perform a first set of rendering tasks for, e.g., UE 14, and UE 14 may complete the rendering process, which may include warping rendered viewport data to correspond to a current view of a user of UE 14. For example, UE 14 may send a predicted pose (position and orientation) of the user to rendering unit 24, and rendering unit 24 may render a viewport according to the predicted pose. However, if the actual pose is different than the predicted pose at the time video data is to be presented to a user of UE 14, UE 14 may warp the rendered data to represent the actual pose (e.g., if the user has suddenly changed movement direction or turned their head).

[0033] While only a single rendering unit is shown in the example of FIG. 1, in other examples, each of UEs 12, 14 may be associated with a corresponding rendering unit. Rendering unit 24 as shown in the example of FIG. 1 is included in AR AS 22, which may be an edge server at an edge of a communication network. However, in other examples, rendering unit 24 may be included in a local network of, e.g., UE 12 or UE 14. For example, rendering unit 24 may be included in a PC, laptop, tablet, or cellular phone of a user, and UE 14 may correspond to a wireless display device, e.g., AR / VR / MR / XR glasses or head mounted display (HMD). Although two UEs are shown in the example of FIG. 1, in general, multi -parti cipant AR calls are also possible.

[0034] UEs 12, 14, and ARAS 22 may communicate AR data using a network communication protocol, such as Real-time Transport Protocol (RTP), which is standardized in Request for Comment (RFC) 3550 by the Internet Engineering Task Force (IETF). These and other devices involved in RTP communications may also implement protocols related to RTP, such as RTP Control Protocol (RTCP), Real-time1616-554WO01Qualcomm Ref. No. 2408048WO 8Streaming Protocol (RTSP), Session Initiation Protocol (SIP), and / or Session Description Protocol (SDP).

[0035] In general, an RTP session may be established as follows. UE 12, for example, may receive an RTSP describe request from, e.g., UE 14. The RTSP describe request may include data indicating what types of data are supported by UE 14. UE 12 may respond to UE 14 with data indicating media streams that can be sent to UE 14, along with a corresponding network location identifier, such as a uniform resource locator (URL) or uniform resource name (URN).

[0036] UE 12 may then receive an RTSP setup request from UE 14. The RTSP setup request may generally indicate how a media stream is to be transported. The RTSP setup request may contain the network location identifier for the requested media data (e.g., media content 64) and a transport specifier, such as local ports for receiving RTP data and control data (e.g., RTCP data) on UE 14. UE 12 may reply to the RTSP setup request with a confirmation and data representing ports of UE 12 by which the RTP data and control data will be sent. UE 12 may then receive an RTSP play request, to cause the media stream to be “played,” i.e., sent to UE 14. UE 12 may also receive an RTSP teardown request to end the streaming session, in response to which, UE 12 may stop sending media data to UE 14 for the corresponding session.

[0037] UE 14, likewise, may initiate a media stream by initially sending an RTSP describe request to UE 12. The RTSP describe request may indicate types of data supported by UE 14. UE 14 may then receive a reply from UE 12 specifying available media streams, such as media content 64, that can be sent to UE 14, along with a corresponding network location identifier, such as a uniform resource locator (URL) or uniform resource name (URN).

[0038] UE 14 may then generate an RTSP setup request and send the RTSP setup request to UE 12. As noted above, the RTSP setup request may contain the network location identifier for the requested media data (e.g., media content 64) and a transport specifier, such as local ports for receiving RTP data and control data (e.g., RTCP data) on UE 14. In response, UE 14 may receive a confirmation from UE 12, including ports of UE 12 that UE 12 will use to send media data and control data.

[0039] After establishing a media streaming session (e.g., AR communication session 28) between UE 12 and UE 14, UE 12 exchange media data (e.g., packets of media data) with UE 14 according to the media streaming session. UE 12 and UE 14 may exchange control data (e.g., RTCP data) indicating, for example, reception statistics by1616-554WO01Qualcomm Ref. No. 2408048WO 9UE 14, such that UEs 12, 14 can perform congestion control or otherwise diagnose and address transmission faults.

[0040] A media streaming session may include audio, video, AR / XR / MR / VR, text, or other such data. The AR / XR / MR / VR data may include three-dimensional objects that a user of a UE device, such as UEs 12, 14, may interact with. For example, UEs 12, 14 may be communicatively coupled to controller devices, e.g., handheld devices including one or more buttons, touchpads, joysticks, gyroscopes, accelerometers, or the like, to receive input. UEs 12, 14 may represent the user’s hands at three dimensional positions in the virtual scene at positions corresponding to the controller devices. The controller devices may allow users of UEs 12, 14 to move through the virtual scene or to interact with three-dimensional objects in the virtual scene. Moreover, the controller devices may include actuators configured to shake, vibrate, or otherwise provide haptic feedback to the user.

[0041] Various devices may be configured to provide various types of haptic feedback. For example, haptic feedback may include vibrotactile, force, tactile, or kinesthetic feedback. Such feedback may be provided by linear resonant actuators (LRAs) and / or eccentric rotating mass (ERM) motors. Additional or alternative haptic feedback may include electrotactile feedback, thermal feedback, surface haptics, ultrasonic haptics, microfluid feedback, and kinesthetic feedback. Likewise, signals for controlling haptic feedback provided by an actuator device (e.g., a device, such as a controller, including an actuator configured to provide haptic feedback) may vary between devices. For example, codepoints may be used to represent various haptic sensations, such as “WAVEFORM CLICK,” “WAVEFORM SUCCESS,” or the like. Waveforms generally correspond to precise patterns of vibration or tactile feedback that may be generated by a haptic actuator to create a particular sensation. Devices may implement and support various waveforms and / or codepoints. Moreover, such codepoints and / or waveforms, and / or other signals for generating haptic feedback, may differ in format, syntax, or the like among various devices.

[0042] Per techniques of this disclosure, UEs 12, 14 may determine whether a controller or other device that can deliver haptic feedback (referred to generally in this disclosure as an “actuator device,” that is, a device that includes an actuator, motor, or the like for providing haptic feedback) is communicatively coupled to UEs 12, 14. A manufacturer of the actuator device may register a type of actuator(s) of the actuator device with a database (generally referred to as an “actuator device database” in this disclosure),1616-554WO01Qualcomm Ref. No. 2408048WO 10 including information to identify the actuator(s) and haptic representations that can be accepted by the actuator(s). The actuator type may be identified by a manufacturer, a product serial number, or the like. Actuators of the same type may have the same identifier value in the actuator device database. The actuator device database may assign a unique actuator type identifier to each type of actuator. The actuator device database may also associate supported haptic representations with each of the actuator type identifiers.

[0043] In this manner, UEs 12, 14 (and / or AR AS 22) may query the actuator device database with an actuator type for controllers or other actuator devices communicatively coupled to UEs 12, 14, to determine what haptic feedback can be delivered and the manner in which to signal the desired haptic feedback. As such, during an AR communication session, when a user interacts with a three-dimensional object in a virtual scene, UEs 12 / 14 (and / or ARAS 22) may render the appropriate signal (e.g., codepoint, vibration pattern, waveform, or the like) for a particular type of supported haptic feedback to deliver that haptic feedback to the user.

[0044] FIG. 2 is a block diagram illustrating an example computing system 100 that may perform split rendering techniques of this disclosure. In this example, computing system 100 includes extended reality (XR) server device 110, network 130, XR client device 140, display device 150, and actuator device 152. XR server device 110 includes XR scene generation unit 112, XR viewport pre-rendering rasterization unit 114, 2D media encoding unit 116, XR media content delivery unit 118, and 5G System (5GS) delivery unit 120.

[0045] Network 130 may correspond to any network of computing devices that communicate according to one or more network protocols, such as the Internet. In particular, network 130 may include a 5G radio access network (RAN) including an access device to which XR client device 140 connects to access network 130 and XR server device 110. In other examples, other types of networks, such as other types of RANs, may be used. For example, network 130 may represent a wireless or wired local network. In other examples, XR client device 140 and XR server device 110 may communicate via other mechanisms, such as Bluetooth, a wired universal serial bus (USB) connection, or the like. XR client device 140 includes 5GS delivery unit 141, tracking / XR sensors 146, XR viewport rendering unit 142, 2D media decoder 144, and XR media content delivery unit 148. XR client device 140 also interfaces with display device 150 to present XR media data to a user (not shown).1616-554WO01Qualcomm Ref. No. 2408048WO 11

[0046] Furthermore, XR client device 140 interfaces with actuator device 152 to deliver haptic feedback to a user. Actuator device 152 may represent a glove, controller, or other device that can deliver haptic feedback to a user. Actuator device 152 may conform to a set of manufacturer proprietary haptic representations and be associated with an identifier (ID) value. The ID value and the set of haptic representations supported by actuator device 152 may be stored by a manufacturer thereof to actuator device database 154.

[0047] Per techniques of this disclosure, XR server device 110 may retrieve the set of haptic representations supported by actuator device 152 from actuator device database 154. For example, XR server device 110 may receive an ID for actuator device 152 from XR client device 140 and query actuator device database 154 with the actuator ID. In response, XR server device 110 may receive the set of haptic representations. XR server device 110 may then obtain support for actuator device 152 and the haptic representations. For example, a user may implement the support as needed, or XR server device 110 may retrieve an implementation of the support from a registry.

[0048] In some examples, XR scene generation unit 112 may correspond to an interactive media entertainment application, such as a video game, which may be executed by one or more processors implemented in circuitry of XR server device 110. XR viewport pre-rendering rasterization unit 114 may format scene data generated by XR scene generation unit 112 as pre-rendered two-dimensional (2D) media data (e.g., video data) for a viewport of a user of XR client device 140. 2D media encoding unit 116 may encode formatted scene data from XR viewport pre-rendering rasterization unit 114, e.g., using a video encoding standard, such as ITU-T H.264 / Advanced Video Coding (AVC), ITU-T H.265 / High Efficiency Video Coding (HEVC), ITU-T H.266 Versatile Video Coding (VVC), or the like. XR media content delivery unit 118 represents a content delivery sender, in this example. In this example, XR media content delivery unit 148 represents a content delivery receiver, and 2D media decoder 144 may perform error handling.

[0049] In general, XR client device 140 may determine a user’s viewport, e.g., a direction in which a user is looking and a physical location of the user, which may correspond to an orientation of XR client device 140 and a geographic position of XR client device 140. Tracking / XR sensors 146 may determine such location and orientation data, e.g., using cameras, accelerometers, magnetometers, gyroscopes, or the like. Tracking / XR sensors 146 provide location and orientation data to XR viewport1616-554WO01Qualcomm Ref. No. 2408048WO 12 rendering unit 142 and 5GS delivery unit 141. XR client device 140 provides tracking and sensor information 132 to XR server device 110 via network 130. XR server device 110, in turn, receives tracking and sensor information 132 and provides this information to XR scene generation unit 112 and XR viewport pre-rendering rasterization unit 114. In this manner, XR scene generation unit 112 can generate scene data for the user’s viewport and location, and then pre-render 2D media data for the user’s viewport using XR viewport pre-rendering rasterization unit 114. XR server device 110 may therefore deliver encoded, pre-rendered 2D media data 134 to XR client device 140 via network 130, e.g., using a 5G radio configuration.

[0050] XR scene generation unit 112 may receive data representing a type of multimedia application (e.g., a type of video game), a state of the application, multiple user actions, or the like. XR viewport pre-rendering rasterization unit 114 may format a rasterized video signal. 2D media encoding unit 116 may be configured with a particular ' er / decoder (codec), bitrate for media encoding, a rate control algorithm and corresponding parameters, data for forming slices of pictures of the video data, low latency encoding parameters, error resilience parameters, intra-prediction parameters, or the like. XR media content delivery unit 118 may be configured with real-time transport protocol (RTP) parameters, rate control parameters, error resilience information, and the like. XR media content delivery unit 148 may be configured with feedback parameters, error concealment algorithms and parameters, post correction algorithms and parameters, and the like.

[0051] Raster-based split rendering refers to the case where XR server device 110 runs an XR engine (e.g., XR scene generation unit 112) to generate an XR scene based on information coming from an XR device, e.g., XR client device 140 and tracking and sensor information 132. XR server device 110 may rasterize an XR viewport and perform XR pre-rendering using XR viewport pre-rendering rasterization unit 114.

[0052] In the example of FIG. 2, the viewport is predominantly rendered in XR server device 110, but XR client device 140 is able to do latest pose correction, for example, using asynchronous time-warping or other XR pose correction to address changes in the pose. XR graphics workload may be split into rendering workload on a powerful XR server device 110 (in the cloud or the edge) and pose correction (such as asynchronous timewarp (ATW)) on XR client device 140. Low motion-to-photon latency is preserved via on-device Asynchronous Time Warping (ATW) or other pose correction methods performed by XR client device 140.1616-554WO01Qualcomm Ref. No. 2408048WO 13

[0053] The various components of XR server device 110, XR client device 140, and display device 150 may be implemented using one or more processors implemented in circuitry, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. The functions attributed to these various components may be implemented in hardware, software, or firmware. When implemented in software or firmware, it should be understood that instructions for the software or firmware may be stored on a computer- readable medium and executed by requisite hardware.

[0054] FIG. 3 is a flowchart illustrating an example method of performing split rendering according to techniques of this disclosure. The method of FIG. 20 is performed by a split rendering client device, such as XR client device 140 of FIG. 2, in conjunction with a split rendering server device, such as XR server device 110 of FIG. 2.

[0055] Initially, the split rendering client device creates an XR split rendering session (200). Creating the XR split rendering session may include any or all of steps 200-208 of FIG. 5, and / or steps 220 and 224 of FIG. 6. As discussed above, creating the XR split rendering session may include, for example, sending device information and capabilities, such as supported decoders, viewport information (e.g., resolution, size, etc.), or the like. The split rendering server device sets up an XR split rendering session (202), which may include setting up encoders corresponding to the decoders and Tenderers corresponding to the viewport supported by the split rendering client device.

[0056] The split rendering client device may then receive current pose and action information (204). For example, the split rendering client device may collect XR pose and movement information from tracking / XR sensors (e.g., tracking / XR sensors 146 of FIG. 2). The split rendering client device may then predict a user pose (e.g., position and orientation) at a future time (206). The split rendering client device may predict the user pose according to a current position and orientation, velocity, and / or angular velocity of the user / a head mounted display (HMD) worn by the user. The predicted pose may include a position in an XR scene, which may be represented as an {X, Y, Z} triplet value, and an orientation / rotation, which may be represented as an {RX, RY, RZ, RW} quaternion value. The split rendering client device may send the predicted pose information, (optionally) along with any actions performed by the user to the split rendering server device (208). For example, the split rendering client device may form1616-554WO01Qualcomm Ref. No. 2408048WO 14 a message according to the format shown in FIG. 8 to indicate the position, rotation, timestamp (indicative of a time for which the pose information was predicted), and optional action information, and send the message to the split rendering server device.

[0057] The split rendering server device may receive the predicted pose information (210) from the split rendering client device. The split rendering server device may then render a frame for the future time based on the predicted pose at that future time (212). For example, the split rendering server device may execute a game engine that uses the predicted pose at the future time to render an image for the corresponding viewport, e.g., based on positions of virtual objects in the XR scene relative to the position and orientation of the user’s pose at the future time. The split rendering server device may then send the rendered frame to the split rendering client device, along with haptic data (214).

[0058] The split rendering client device may then receive the rendered frame and the haptic data (216) and present the rendered frame at the future time (218). For example, the split rendering client device may receive a stream of rendered frames and store the received rendered frames to a frame buffer. At a current display time, the split rendering client device may determine the current display time and then retrieve one of the rendered frames from the buffer having a presentation time that is closest to the current display time. The split rendering client device may also deliver haptic feedback (220) via an actuator device.

[0059] FIG. 4 is a conceptual diagram illustrating a system 250 for transmitting XR media data including haptic representations. In this example, system 250 includes encoder 252, decoder 254, and Tenderer 256. Encoder 252 encodes data related to haptic representations. In MPEG per ISO / IEC 23090-31, haptic representations may be coded as descriptive representations or pulse code modulation (PCM) representations. Encoder 252 may transcode from existing descriptive representations (e.g., .ivs or .ahap files) to a common format, e.g., .hjif. This transcoding may be according to a compression scheme that is based on frequency band decomposition, keyframe extraction, and wavelet encoding for PCM representation (.wav). Encoder 252 may send the common format for the haptic representations to decoder 254. Decoder 254 may decode the haptic representations and provide decoded haptic representations to Tenderer 256 to perform haptic feedback.

[0060] FIG. 5 is a flow diagram illustrating an example actuator registration process according to techniques of this disclosure. Initially, a manufacturer of actuator device1616-554WO01Qualcomm Ref. No. 2408048WO 15280 (e.g., a glove), such as actuator device 152 of FIG. 2, registers the actuator device to actuator device database 282, such as actuator device database 154 of FIG. 2, and supported haptic representations to actuator device database 282 (290). This registration may include registration of an actuator type ID and haptic representations supported by actuator device 280. In particular, the registration may include information identifying actuator device 280 and haptic representations that can be accepted by actuators of actuator device 280.

[0061] Haptic representations may be codepoints of a self-defined internal code with associated semantics. For example, “00” may represent light vibration, “01” may represent medium vibration, “10” may represent strong vibration, “11” may represent extreme vibration. This gives the manufacturer the flexibility to change the waveform if required or upgrade the waveform without affecting the representation.

[0062] The actuator ID may be any sort of identification of actuator device 280, such as data including manufacturer, device model number, device lot number, device serial number, barcode, QR code, or the like. In general, all actuator devices of the same type may have the same identifier. Actuator device database 282 may assign a unique actuator type ID to the type of actuators. Actuator device database 282 may assign additional actuator type IDs representing which of the additional associated haptic representations are accepted by the type of actuators. This allows an augmented reality (AR) device, such as AR device 284, participating in an AR communication session (e.g., a split rendering server such as XR server device 110 of FIG. 2 or a client device such as XR client device 140) to take advantage of backward compatibility of the actuator. For example, a new actuator may support a new haptic representation and older haptic representations, but AR device 284 may only support the old haptic representations. In this case, AR device 284 may use the old haptic representations.

[0063] AR device 284 may then send a query specifying the actuator ID to actuator device database 282 (292). In response, AR device 284 may receive the haptic representations supported by the actuator type corresponding to the actuator ID from actuator device database 282 (294). AR device 284 may be a server, such as a cloud server, a local device, or other device involved in the AR communication session, e.g., for split rendering (as a split rendering server or split rendering client). In any case, AR device 284 may query the actuator device database for registered types of actuators and for the respective supported haptic representations. For each of the types of actuators, the haptic representations received in response to the query may include a list of1616-554WO01Qualcomm Ref. No. 2408048WO 16 channels, waveforms, codepoints, and / or a corresponding haptic representation. As an example, a vest wearable by a user / player may include a plurality of actuators, each corresponding to a respective channel. The haptic representation may be a PCM waveform format, a descriptive representation, or an actuator defined code, which may be standardized or proprietary. Haptic representations may be provided for each of the channels in the list.

[0064] A user of AR device 284 may then implement support for the actuator device and the haptic representation(s) (296). For example, the user may configure AR device 284 to convert haptic media in an internal haptic representation of AR device 284 to haptic media in the supported haptic representations for a registered type of actuator.

[0065] FIG. 6 is a flow diagram illustrating an example technique for retrieving actuator information to generate actuator-type-specific haptic representations according to techniques of this disclosure. In this example, various devices participate, including actuator device database 300, actuator device 302 (e.g., a glove or controller), client device 304 (e.g., XR client device 140 of FIG. 2 or other user equipment (UE) device), and a split rendering server 306 (e.g., XR server device 110 of FIG. 2). In this example, split rendering server 306 generates actuator-type-specific haptic representations.

[0066] Initially, client device 304, such as display device 150 or XR client device 140 of FIG. 2, requests an actuator identifier for actuator device 302 (e.g., actuator device 152 of FIG. 2) (310). In response, actuator device 302 sends the actuator identifier to client device 304, which receives the actuator identifier for actuator device 302 (312). Client device 304 then requests actuator information from actuator device database 300 (e.g., actuator device database 154 of FIG. 2) for the actuator identifier of actuator device 302 (314). Client device 304 may also request corresponding haptic representations for actuators of actuator device 302 from actuator device database 300. Client device 304 the receives actuator information for actuator device 302 from actuator device database 300 (316), which may include haptic representations supported by actuator device 302. Alternatively, client device 304 may first query retrieve actuator information from actuator device database 300, determine the actuator identifier for actuator device 302, then determine the supported haptic representations using the actuator identifier from the actuator information.

[0067] Client device 304 in this case then negotiates actuator support with split rendering server 306 (318), e.g., using data of a session description protocol (SDP) message, such as an SDP Offer or SDP Answer. This SDP message may be exchanged1616-554WO01Qualcomm Ref. No. 2408048WO 17 during session startup. When actuator device 302 has multiple actuator IDs (e.g., when actuator device 302 supports haptic representations of old actuators), client device 304 and split rendering server 306 may determine which of the actuator IDs to use for the session, e.g., the highest ID that is supported, representing the newest supported haptic representation, or the like.

[0068] FIG. 7 is a flow diagram illustrating an example technique for generating actuator-type-specific haptic representations per techniques of this disclosure. During an XR media session, client device 304 may send pose data for a user to split rendering server 306 (320). Split rendering server 306 may then update the scene (322) based on the user’s pose (and other input data, such as movement of other participants, interactions with virtual objects, or the like). Split rendering server 306 may also generate video, audio, and haptic output to be sent to client device 304 (324). Split rendering server 306 may convert the haptic output into haptic information (e.g., haptic media) according to the negotiated, supported representation of actuator device 302 associated with client device 304 (326). Split rendering server 306 may then send media data including audio, video, and the haptic information to Client device 304 (328).

[0069] Client device 304 may then extract the haptic information and forward the haptic information to actuator device 302 (330). Actuator device 302 may then use the received haptic information to synthesize haptic sensations (332) synchronously with presentation of the audio and video data by client device 304.

[0070] FIG. 8 is a flowchart illustrating an example method of communicating media data, e.g., as part of an augmented reality (AR) media communication session, per techniques of this disclosure. The method of FIG. 8 may be performed by a device involved in an AR media communication session, such as a client device (e.g., a UE device, such as one of UEs 12, 14 of FIG. 1, XR client device 140 of FIG. 2, AR device 284 of FIG. 5, or client device 304 of FIGS. 6 and 7) or a split rendering server or other server device, such as a cloud server device (e.g., AR AS 22 of FIG. 1, XR server device 110 of FIG. 2, or split rendering server 306 of FIGS. 6 and 7). For purposes of example and explanation, the method of FIG. 8 is explained with respect to AR device 284 of FIG. 5.

[0071] Initially, AR device 284 establishes an AR communication session, e.g., with another client device (e.g., AR communication session 28 between UEs 12 and 14 of FIG. 1). Such session establishment may involve exchanging SDP offer / answer1616-554WO01Qualcomm Ref. No. 2408048WO 18 messages, which may indicate that haptic feedback is supported for the AR communication session. AR device 284 may be associated with an actuator device, such as a controller, glove, or other device capable of delivering haptic feedback (e.g., via an actuator, motor, or the like).

[0072] Thus, AR device 284 may determine an actuator identifier for the actuator device (350). For example, AR device 284 may query the actuator device directly or receive data representing the actuator identifier via another device involved in the AR communication session (e.g., a split rendering server may receive an actuator identifier from a client device, such as a UE device, that is in communication with an actuator device). AR device 284 may then query an actuator device database using the actuator identifier (352). In response, AR device 284 may receive information representing haptic representations supported by the actuator device of the actuator type indicated by the actuator identifier (354).

[0073] AR device 284 may then receive data as part of the AR communication session indicating haptic sensations to be presented by the actuator device (356). The data representing the haptic sensations may accompany other data to be presented as part of the AR communication session, e.g., audio data, video data, an indication of one or more avatars and animation streams for animating the avatars, or the like. AR device 284 may determine a supported haptic representation corresponding to the haptic sensation (358). AR device 284 may then send data indicating the haptic representation to be presented to the actuator device (360), e.g., directly or via another device, such as via a UE device in communication with the actuator device.

[0074] In this manner, the method of FIG. 8 represents an example of a method of communicating media data including: retrieving, by a device participating in an augmented reality (AR) media session, data from an actuator device database representative of one or more haptic representations supported by an actuator device associated with the device participating in the AR media session; and sending, by the device participating in the AR media session, haptic information as part of the AR media session, the haptic information conforming to one of the one or more haptic representations supported by the actuator device.

[0075] Various examples of the techniques of this disclosure are summarized in the following clauses:

[0076] Clause 1: A method of communicating media data, the method comprising: retrieving, by a device participating in an extended reality (XR) media session, data1616-554WO01Qualcomm Ref. No. 2408048WO 19 from an actuator device database representative of one or more haptic representations supported by an actuator device associated with the device participating in the XR media session; and exchanging, by the device participating in the XR media session, haptic information as part of the XR media session, the haptic information conforming to one of the one or more haptic representations supported by the actuator device.

[0077] Clause 2: The method of clause 1, wherein sending the haptic information comprises sending the haptic information along with audio, video, graphical, or other media data of the XR media session.

[0078] Clause 3: The method of any of clauses 1 and 2, wherein the haptic representations include codepoints associated with semantics.

[0079] Clause 4: The method of clause 3, wherein the semantics represent at least one of vibration intensity, vibration pattern, or vibration waveform.

[0080] Clause 5: The method of any of clauses 1-4, wherein retrieving the data from the actuator device database comprises querying the actuator device database using an identifier value for the actuator device.

[0081] Clause 6: The method of clause 5, wherein the identifier value comprises a serial number for the actuator device.

[0082] Clause 7: The method of any of clauses 5 and 6, wherein the identifier value includes data representing a manufacturer of the actuator device.

[0083] Clause 8: The method of any of clauses 5-7, wherein the actuator device conforms to a type of actuator device, and wherein the identifier value includes data representative of the type of actuator device.

[0084] Clause 9: The method of any of clauses 1-8, wherein the one or more haptic representations each include data representing a list of channels and, for each of the channels, data representing a haptic representation for the channel.

[0085] Clause 10: The method of clause 9, wherein the data representing the haptic representation for the channel comprises one or more of a pulse code modulation (PCM) waveform format, a descriptive representation, or an actuator defined code.

[0086] Clause 11: The method of any of clauses 1-10, wherein the device participating in the XR media session comprises a server device.

[0087] Clause 12: The method of clause 11, wherein the server device comprises a cloud server device.

[0088] Clause 13: The method of clause 11, wherein the server device comprises a split rendering server device.1616-554WO01Qualcomm Ref. No. 2408048WO 20

[0089] Clause 14: The method of any of clauses 11-13, wherein retrieving the data comprises: querying, by the server device, the actuator device database using an identifier value for the actuator device; and receiving, by the server device, the one or more haptic representations supported by the actuator device.

[0090] Clause 15: The method of any of clauses 1-10, wherein the device participating in the XR media session comprises a user equipment (UE) device.

[0091] Clause 16: The method of clause 15, wherein retrieving the data comprises: retrieving, by the UE device, one or more actuator identifier values from the actuator device; and querying, by the UE device, the actuator device database using the one or more actuator identifier values to retrieve the one or more haptic representations.

[0092] Clause 17: The method of any of clauses 15 and 16, further comprising negotiating actuator support with a split rendering device.

[0093] Clause 18: The method of clause 17, wherein negotiating support comprises exchanging a session description protocol (SDP) message including data representing the one or more haptic representations.

[0094] Clause 19: The method of any of clauses 15-18, wherein exchanging the haptic information comprises receiving XR media data including the haptic information from a split rendering device.

[0095] Clause 20: A device for communicating media data, the device comprising one or more means for performing the method of any of clauses 1-20.

[0096] Clause 21 : The system of clause 4, wherein the one or more means comprise a processing system comprising one or more processors implemented in circuitry, and a memory configured to store XR media data.

[0097] Clause 22: A device for participating in an extended reality (XR) media session, the device comprising: means for retrieving data from an actuator device database representative of one or more haptic representations supported by an actuator device associated with the device for participating in the XR media session; and means for exchanging haptic information as part of the XR media session, the haptic information conforming to one of the one or more haptic representations supported by the actuator device.

[0098] Clause 23: The device of clause 22, wherein the device comprises a user equipment (UE) device.

[0099] Clause 24: The device of clause 22, wherein the device comprises a split rendering server device.1616-554WO01Qualcomm Ref. No. 2408048WO 21

[0100] Clause 25: A method of communicating media data, the method comprising: retrieving, by a device participating in an augmented reality (AR) media session, data from an actuator device database representative of one or more haptic representations supported by an actuator device associated with the device participating in the AR media session; and sending, by the device participating in the AR media session, haptic information as part of the AR media session, the haptic information conforming to one of the one or more haptic representations supported by the actuator device.

[0101] Clause 26: The method of clause 25, wherein sending the haptic information comprises sending the haptic information along with audio, video, graphical, or other media data of the AR media session.

[0102] Clause 27: The method of clause 25, wherein the haptic representations include codepoints associated with semantics.

[0103] Clause 28: The method of clause 27, wherein the semantics represent at least one of vibration intensity, vibration pattern, or vibration waveform.

[0104] Clause 29: The method of clause 25, wherein retrieving the data from the actuator device database comprises querying the actuator device database using an identifier value for the actuator device.

[0105] Clause 30: The method of clause 29, wherein the identifier value comprises a serial number for the actuator device.

[0106] Clause 31: The method of clause 29, wherein the identifier value includes data representing a manufacturer of the actuator device.

[0107] Clause 32: The method of clause 29, wherein the actuator device conforms to a type of actuator device, and wherein the identifier value includes data representative of the type of actuator device.

[0108] Clause 33: The method of clause 25, wherein the one or more haptic representations each include data representing a list of channels and, for each of the channels, data representing a haptic representation for the channel.

[0109] Clause 34: The method of clause 33, wherein the data representing the haptic representation for the channel comprises one or more of a pulse code modulation (PCM) waveform format, a descriptive representation, or an actuator defined code.

[0110] Clause 35: The method of clause 25, wherein the device participating in the AR media session comprises a server device.

[0111] Clause 36: The method of clause 35, wherein the server device comprises a cloud server device.1616-554WO01Qualcomm Ref. No. 2408048WO 22

[0112] Clause 37: The method of clause 35, wherein the server device comprises a split rendering server device.

[0113] Clause 38: The method of clause 35, wherein retrieving the data comprises: querying, by the server device, the actuator device database using an identifier value for the actuator device; and receiving, by the server device, the one or more haptic representations supported by the actuator device.

[0114] Clause 39: The method of clause 25, wherein the device participating in the AR media session comprises a user equipment (UE) device.

[0115] Clause 40: The method of clause 39, wherein retrieving the data comprises: retrieving, by the UE device, one or more actuator identifier values from the actuator device; and querying, by the UE device, the actuator device database using the one or more actuator identifier values to retrieve the one or more haptic representations.

[0116] Clause 41: The method of clause 39, further comprising negotiating actuator support with a split rendering device.

[0117] Clause 42: The method of clause 41, wherein negotiating support comprises exchanging a session description protocol (SDP) message including data representing the one or more haptic representations.

[0118] Clause 43: The method of clause 39, further comprising receiving AR media data including the haptic information from a split rendering device.

[0119] Clause 44: A device for participating in an augmented reality (AR) media session, the device being associated with an actuator device, the device comprising: a memory configured to store media data; and a processing system implemented in circuitry and configured to: retrieve data from an actuator device database representative of one or more haptic representations supported by the actuator device; and send haptic information as part of the AR media session, the haptic information conforming to one of the one or more haptic representations supported by the actuator device.

[0120] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer-1616-554WO01Qualcomm Ref. No. 2408048WO 23 readable media generally may correspond to (1) tangible computer-readable storage media which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0121] By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are instead directed to non-transitory, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0122] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated in a combined codec. Also, the techniques could be fully implemented in one or more circuits or logic elements.1616-554WO01Qualcomm Ref. No. 2408048WO 24

[0123] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a codec hardware unit or provided by a collection of interoperative hardware units, including one or more processors as described above, in conjunction with suitable software and / or firmware.

[0124] Various examples have been described. These and other examples are within the scope of the following claims.1616-554WO01

Claims

Qualcomm Ref. No. 2408048WO 25WHAT IS CLAIMED IS:

1. A method of communicating media data, the method comprising: retrieving, by a device participating in an augmented reality (AR) media session, data from an actuator device database representative of one or more haptic representations supported by an actuator device associated with the device participating in the AR media session; and sending, by the device participating in the AR media session, haptic information as part of the AR media session, the haptic information conforming to one of the one or more haptic representations supported by the actuator device.

2. The method of claim 1, wherein sending the haptic information comprises sending the haptic information along with audio, video, graphical, or other media data of the AR media session.

3. The method of claim 1, wherein the haptic representations include codepoints associated with semantics.

4. The method of claim 3, wherein the semantics represent at least one of vibration intensity, vibration pattern, or vibration waveform.

5. The method of claim 1, wherein retrieving the data from the actuator device database comprises querying the actuator device database using an identifier value for the actuator device.

6. The method of claim 5, wherein the identifier value comprises a model number for the actuator device.

7. The method of claim 5, wherein the identifier value includes data representing a manufacturer of the actuator device.

8. The method of claim 5, wherein the actuator device conforms to a type of actuator device, and wherein the identifier value includes data representative of the type of actuator device.1616-554WO01Qualcomm Ref. No. 2408048WO 269. The method of claim 1, wherein the one or more haptic representations each include data representing a list of channels and, for each of the channels, data representing a haptic representation for the channel.

10. The method of claim 9, wherein the data representing the haptic representation for the channel comprises one or more of a pulse code modulation (PCM) waveform format, a descriptive representation, or an actuator defined code.

11. The method of claim 1 , wherein the device participating in the AR media session comprises a server device.

12. The method of claim 11, wherein the server device comprises a cloud server device.

13. The method of claim 11, wherein the server device comprises a split rendering server device.

14. The method of claim 11, wherein retrieving the data comprises: querying, by the server device, the actuator device database using an identifier value for the actuator device; and receiving, by the server device, the one or more haptic representations supported by the actuator device.

15. The method of claim 1, wherein the device participating in the AR media session comprises a user equipment (UE) device.

16. The method of claim 15, wherein retrieving the data comprises: retrieving, by the UE device, actuator information from the actuator device; and querying, by the UE device, the actuator device database using the actuator information to retrieve the one or more haptic representations.1616-554WO01Qualcomm Ref. No. 2408048WO 2717. The method of claim 15, further comprising negotiating actuator support with a split rendering device.

18. The method of claim 17, wherein negotiating support comprises exchanging a session description protocol (SDP) message including data representing the one or more haptic representations.

19. The method of claim 15, further comprising receiving AR media data including the haptic information from a split rendering device.

20. A device for participating in an augmented reality (AR) media session, the device being associated with an actuator device, the device comprising: a memory configured to store media data; and a processing system implemented in circuitry and configured to: retrieve data from an actuator device database representative of one or more haptic representations supported by the actuator device; and send haptic information as part of the AR media session, the haptic information conforming to one of the one or more haptic representations supported by the actuator device.1616-554WO01

Images