ISOBMFF haptic track with sample anchoring for haptic effects

Abstract

A method, apparatus, and system for haptic signal processing are provided. The processing can include receiving a media stream including at least one haptic track and at least one video track. The processing can include obtaining one or more Moving Picture Experts Group (MPEG) Immersive Haptic Stream (MIHS) units from the media stream and obtaining timing information associated with one or more haptic effects from the media stream, the timing information including at least one temporal position of the one or more haptic effects. The processing can then include rendering the media stream based on the obtained timing information.

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Patent Application No. 63 / 416,780, filed October 17, 2022, and U.S. Patent Application No. 18 / 487,688, filed October 16, 2023, the disclosures of which are incorporated herein by reference in their entireties.

[0002] [Technical field] This disclosure relates to a set of advanced video coding techniques, and more particularly to encoding and decoding haptic experiences for multimedia presentations. [Background technology]

[0003] Haptic experiences are already part of multimedia presentations. In applications where a multimedia presentation includes aspects of a haptic experience, haptic signals are transmitted to a device or wearable device, allowing the user to feel haptic sensations that coordinate with the visual and / or audio media experience while using the application.

[0004] Recognizing that haptic experiences are becoming increasingly popular in multimedia presentations, the Motion Picture Experts Group (MPEG) has already begun work on compression standards for haptics (both MPEG-DASH and MPEG-I) and signaling transport for compressed haptics in the International Organization for Standardization (ISO) Based Media File Format (ISOBMFF). Summary of the Invention [Problem to be solved by the invention]

[0005] One of the problems to be solved regarding the haptic experience aspect within a multimedia presentation is that the timing model for the transmission of haptic tracks is unclear, i.e., it is unclear how the timing of the ISOBMFF tracks relates to the timing of the haptic base signals. A solution is needed to address this problem. [Means for solving the problem]

[0006] According to an embodiment, there is provided a method for encoding or decoding haptic data, the method being executable by at least one processor and including the steps of: receiving a media stream including at least one haptic track and at least one video track; obtaining one or more Motion Experts Group (MPEG) Immersive Haptics Stream (MIHS) units from the media stream, the MIHS units including one or more haptic effects and including start times; obtaining timing information associated with the one or more haptic effects from the media stream, the timing information including at least one temporal position of the one or more haptic effects; and rendering the media stream based on the obtained timing information.

[0007] According to an embodiment, an apparatus for haptic processing may be provided, the apparatus including at least one memory configured to store program code and at least one processor configured to read the program code and operate according to instructions of the program code, the program code may include: first receiving code configured to cause the at least one processor to receive a media stream including at least one haptic track and at least one video track; first acquiring code configured to cause the at least one processor to acquire one or more Motion Experts Group (MPEG) Immersive Haptic Stream (MIHS) units from the media stream, the MIHS units including one or more haptic effects, the MIHS units including start times; second acquiring code configured to cause the at least one processor to acquire timing information associated with the one or more haptic effects from the media stream, the timing information including at least one temporal position of the one or more haptic effects; and rendering code configured to cause the at least one processor to render the media stream based on the acquired timing information.

[0008] According to an embodiment, a non-transitory computer-readable medium may be provided having stored thereon computer instructions, which may include one or more instructions that, when executed by one or more processors of an apparatus for haptic processing, cause the one or more processors to: receive a media stream including at least one haptic track and at least one video track; obtain one or more Motion Experts Group (MPEG) Immersive Haptic Stream (MIHS) units from the media stream, the MIHS units including one or more haptic effects and including start times; obtain timing information associated with the one or more haptic effects from the media stream, the timing information including at least one temporal position of the one or more haptic effects; and render the media stream based on the obtained timing information.

[0009] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a schematic diagram of a simplified block diagram of a streaming system according to an embodiment of the present disclosure. [Figure 3A] FIG. 1 is a schematic diagram of a simplified block diagram of a haptic encoder according to an embodiment of the present disclosure. [Figure 3B] FIG. 1 is a schematic diagram of a simplified block diagram of a haptic decoder and a haptic renderer according to an embodiment of the present disclosure. [Figure 4] 1 is an exemplary diagram of a process for determining relative timing of MPEG immersive haptic stream (MIHS) units according to an embodiment of the present disclosure. FIG. [Figure 5]FIG. 10 is an exemplary diagram of a process for determining relative timing of MIHS units according to an embodiment of the present disclosure. [Figure 6] 1 is an exemplary flowchart of a process for processing haptic media according to an embodiment of the present disclosure. [Figure 7] FIG. 1 illustrates a computer system suitable for implementing embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0011] According to one aspect of the present disclosure, there is provided a method, system, and non-transitory storage medium for parallel processing of dynamic mesh compression. Embodiments of the present disclosure may also be applied to static meshes.

[0012] 1 and 2, an embodiment of the present disclosure for realizing the encoding and decoding configurations of the present disclosure will be described.

[0013] 1 shows a simplified block diagram of a communication system 100 according to an embodiment of the present disclosure. The system 100 may include at least two terminals 110, 120 interconnected via a network 150. In the case of one-way data transmission, the first terminal 110 may encode video data, which may include mesh data, at a local location and transmit the encoded video data to the other terminal 120 via the network 150. The second terminal 120 may receive the encoded video data of the other terminal from the network 150, decode the encoded data, and display the restored video data. One-way data transmission is common in media service applications, etc.

[0014] 1 illustrates a second set of terminals 130, 140 provided to support two-way transmission of encoded video, such as might occur during a video conference. For two-way transmission of data, each terminal 130, 140 can encode video data captured at a local location and transmit it to another terminal over network 150. Each terminal 130, 140 can receive the encoded video data transmitted by the other terminal, decode the encoded data, and display the recovered video data on a local display device.

[0015] In FIG. 1 , terminals 110-140 may be, for example, servers, personal computers, smartphones, and / or any other type of terminal. For example, terminals 110-140 may be laptop computers, tablet computers, media players, and / or dedicated videoconferencing devices. Network 150 represents any number of networks transmitting coded video data between terminals 110-140, including, for example, wired and / or wireless communication networks. Communication network 150 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, unless otherwise described below, the architecture and topology of network 150 may not be important to the operation of the present disclosure.

[0016] As an example application of the disclosed subject matter, Figure 2 illustrates the placement of a video encoder and decoder in a streaming environment. The disclosed subject matter can be used with other applications that support video, including video conferencing, digital TV, and the like, which store compressed video on digital media such as Compact Discs (CDs), Digital Video Disks (DVDs), memory sticks, etc.

[0017] 2, the streaming system 200 includes a capture subsystem 213 that includes a video source 201 and an encoder 203. The streaming system 200 may further include at least one streaming server 205 and / or at least one streaming client 206.

[0018] The video source 201 can create a stream 202 including, for example, a three-dimensional (3D) mesh and metadata associated with the 3D mesh. The video source 201 can include, for example, a 3D sensor (e.g., a depth sensor) or 3D imaging technology (e.g., a digital camera) and a computing device configured to generate a 3D mesh using data received from the 3D sensor or 3D imaging technology. The sample stream 202, which has a high amount of data compared to an encoded video bitstream, can be processed by an encoder 203 coupled to the video source 201. The encoder 203 can include hardware, software, or a combination thereof to realize or implement aspects of the disclosed subject matter, as described in detail below. The encoder 203 can generate an encoded video bitstream 204. The encoded video bitstream 204, which has a low amount of data compared to the uncompressed stream 202, can be stored on a streaming server 205 for future use. One or more streaming clients 206 can access the streaming server 205 to retrieve a video bitstream 209, which can be a copy of the encoded video bitstream 204.

[0019] The streaming client 206 may include a video decoder 210 and a display 212. The video decoder 210 may decode a video bitstream 209, which may be, for example, an input copy of the encoded video bitstream 204, and generate a transmit video sample stream 211 that may be rendered on the display 212 or another rendering device (not shown). In some streaming systems, the video bitstreams 204, 209 may be encoded according to some video encoding / compression standard.

[0020] 3A-3B, an embodiment of the present disclosure implementing a haptic encoder 300 and a haptic decoder 350 will be described.

[0021] 3A, the haptic encoder 300 can receive both descriptive haptic data and waveform haptic data. Thus, the haptic encoder 300 can receive three types of input files: .ohm metadata files (object haptic metadata, a text file format used for haptic metadata), descriptive haptic files (.ivs, .ahap, and .hjif), or waveform PCM (Pulse Code Modulation) files. The system can process PCM (Pulse Code Modulation) files (.wav). Examples of descriptive data include .ahap (Apple Haptic and Audio Modes - a file format similar to JSON (JavaScript Object Notation, JSON) specifying haptic patterns) from Apple (representing the expected haptic output with a parameterized set of modulated continuous signals and a set of modulated transients), .ivs from Immersion (representing the expected haptic output with a set of basic effects parameterized by a set of parameters), or the submitted MPEG format .hjif (Haptic JSON Interchange Format). Examples of waveform Pulse Code Modulation (PCM) signals can include .ohm input files containing metadata information.

[0022] According to an embodiment, the haptic encoder 300 can process two types of input files differently: For descriptive content, the haptic encoder 300 can semantically analyze the input and transcode the data (if necessary) into a submitted encoded representation.

[0023] According to an embodiment, the .ohm metadata input file may include a description of the haptic system and installation. In particular, the .ohm metadata input file may include the name of each associated haptic file (descriptive or PCM) and a description of the signal. It may also provide a mapping between each channel of the signal and a target body part on the user's body. For the .ohm metadata input file, the haptic encoder performs metadata extraction by retrieving the associated haptic file from a Uniform Resource Identifier (URI), encoding the haptic file based on its type, extracting metadata from the .ohm file, and mapping the metadata to metadata information in the data model.

[0024] According to an embodiment, descriptive haptic files (e.g., .ivs, .ahap, and .hjif) can be encoded using a simple process. The haptic encoder 300 first specifically identifies the input format. If the input format is an .hjif file, no transcoding is required; the file can be further edited, compressed into a binary format, and finally packetized into an MIHS stream. If an .ahap or .ivs input file is used, transcoding is required. The haptic encoder 300 first semantically analyzes the input file information, then transcodes and formats it into the selected data model. After being transcoded, the data is exported to an .hjif file, an .hmpg binary file, or an MIHS stream.

[0025] According to an embodiment, the haptic encoder 300 can perform signal analysis to describe the signal structure of a .wav file and convert the signal structure into the submitted coded representation. For waveform PCM content, the haptic encoder 300 can divide the signal analysis process into two sub-processes. After performing frequency band decomposition on the signal, the first sub-process can encode the low-frequency band using a keyframe extraction process. The low-frequency band can then be reconstructed and the error between the signal and the original low-frequency signal can be calculated. This residual signal is then added to the original high-frequency band, which is then coded using a wavelet transform, which is the second sub-process. According to an embodiment, when several low-frequency bands are used, the residuals from all low-frequency bands are added to the high-frequency band before coding. In an embodiment, when several high-frequency bands are used, the residuals from the low-frequency bands are added to the first high-frequency band before coding.

[0026] According to an embodiment, keyframe extraction involves extracting low-frequency bands from frequency band decomposition and analyzing their content in the time domain. According to an embodiment, wavelet processing involves extracting high-frequency bands from frequency band decomposition and the low-frequency residual and dividing the high-frequency bands into equal-sized blocks. These equal-sized signal blocks are then analyzed in a psycho-tactile model. With the help of the psycho-tactile model, lossy compression can be applied by wavelet transforming and quantizing the blocks. Finally, each block is stored as a separate effect within a single band, which is performed during formatting. Binary compression can be applied using lossless compression using appropriate coding techniques, such as the Set Partitioning in Hierarchical Trees (SPIHT) algorithm and Arithmetic Coding (AC).

[0027] As shown in FIG. 3A, a haptic encoder 300 is configured to encode descriptive haptic data and quantized haptic data and can output three types of formats: an interchange format (.hjif), a binary compressed format (.hmpg), and a stream format (e.g., MPEG Immersive Haptic Stream (MIHS)). The .hjif format is a human-readable format based on JSON, allowing for easy parsing and manual editing, making it an ideal interchange format, especially for content design / creation. For distribution purposes, the .hjif data can be compressed into a more memory-efficient binary .hmpg bitstream. Such compression is lossy, and different parameters affect the coding depth of the amplitude and frequency that make up the bitstream. For streaming purposes, the data is compressed and packetized into an MPEG-I Haptic Stream (MIHS). The above three formats have complementary purposes, and lossy one-to-one conversions can be performed between them.

[0028] As shown in Figure 3B, the haptic decoder 350 can use the .hmpg compressed binary file format or an MIHS bitstream as input. The haptic decoder 350 can output the .hjif interchange format, which can be used directly for rendering. Both types of input formats can be decompressed binary to extract both the metadata and the data itself from the file and map the data into a selected data structure. The data can then be delivered to the haptic renderer 380 in the .hjif format.

[0029] As shown in FIG. 3B, the renderer 380 includes a compositor. The compositor can render haptic data from an .hjif input file into a PCM output file. The rendering and / or compositing is informative. According to an embodiment, the compositor analyzes the input file and performs high-level compositing distributions between vectors, wavelets, etc. The compositing process then continues down to the band components of the codec that invoke the compositing process. A simple adder then mixes all bands of a given channel to recreate the desired haptic signal.

[0030] According to an embodiment, a haptic experience defines the root of a hierarchical data model: it provides information about the file date and format version, describes the haptic experience, lists the different avatars (i.e., body representations) used throughout the experience, and defines all haptic perceptions.

[0031] According to an embodiment, a self-contained stream format for transmitting MPEG-I haptic data can use a packetization approach and can include two levels of packets: an MPEG-I haptic stream (MIHS) unit that covers a duration and includes zero or more MIHS packets, and an MIHS packet that contains metadata or haptic effect data. Each MIHS unit can cover a non-overlapping duration of haptic presentation time, i.e., each MIHS unit can start at the end of the previous MIHS unit and cover a duration bounded by its duration field. An MIHS unit may contain a subsequent MIHS unit as long as it is not the last MIHS unit of the haptic experience. All MIHS packets of an MIHS unit can have a start time and a duration that the MIHS unit is included in.

[0032] In an embodiment, the MIHS unit is also referred to as a network abstraction layer unit associated with haptic data.In an embodiment, the MIHS unit is also referred to as an MIHS sample associated with haptic data.

[0033] According to the embodiment, MIHS units may be synchronous or asynchronous. Synchronous units reset previous effects to provide a haptic experience independent of the previous MIHS unit. Asynchronous units are a continuation of the previous MIHS unit and cannot be decoded and rendered independently without decoding the previous MIHS unit.

[0034] According to embodiments, multiple channels can encode haptic signals. In some embodiments, a haptic channel can limit the signal rendered at a specific body position by a dedicated actuator / device. Metadata stored at the channel level can include, for example, the gain associated with the channel, blend weights, the desired body position of the haptic feedback, and selectable reference devices and / or directions. Additional information, such as a desired sampling frequency and sample count, can also be provided. Finally, the haptic data for a channel is contained in a set of haptic bands, which are limited by the frequency range of the haptic band. A haptic band describes the haptic signal for a channel in a given frequency range. A band is limited by the type and ordered list of haptic effects, with each haptic effect containing a set of keyframes. For each haptic band type, the haptic effects can be limited by at least the position and type. The position can indicate the temporal or spatial location of the effect. In some embodiments, a value of 0 is the relative starting position of the experience and depends on the factor variables of the deployed sensory modality. The default unit for temporal haptic feedback may be milliseconds, and the default unit for spatial haptic feedback may be millimeters. The present embodiment discloses the "start position of the experience" because the binary delivery format does not have any concept of a bounded time interval, ie a frame or a sample.

[0035] Depending on the type of band and the type of effect, additional characteristics can be specified, including the phase, the base signal, and the composition and number of consecutive haptic keyframes that describe the effect.

[0036] According to an embodiment, the present disclosure limits the haptic data hierarchy. ●Tactile Channel ○Tactile band ■Tactile effect

[0037] Embodiments of the present disclosure describe two anchors for the location of a haptic effect relative to an ISOBMFF track.

[0038] Figure 4 shows a first embodiment. As shown in Figure 4, each MIHS unit (also referred to in the embodiment as an MIHS sample, an ISOBMFF haptic sample, or a sample) includes one or more haptic channel information and one or more haptic band information. As described above, each MIHS unit includes one or more channels, each channel includes one or more bands, and each band may have one or more effects.

[0039] In a first embodiment, the temporal position of the effect can be defined as an offset relative to the start timing (e.g., MIHS unit start time) of the sample that carries the effect. In a second or the same embodiment, the offset is based on the start time and / or presentation time of the media or haptic track.

[0040] According to an embodiment, the first embodiment can manipulate the tracks if it does not affect the position of the haptic effect, since any changes in ISOBMFF sample timing do not affect the relative position of the effect. According to an embodiment, for basic haptic streams (e.g., high-level grammar streams), the second embodiment can be used when using basic streams without ISOBMFF.

[0041] According to an embodiment, several types of haptic tracks can be used. In an embodiment, a haptic track can use samples or MIHS units with temporal positions of effects, where the temporal position of the effect is defined as an offset relative to the start timing of the sample. According to another embodiment, a haptic track uses samples or MIHS units whose temporal position of the effect is relative to the start time of the track. In another embodiment, mixed MIHS units or samples can be used.

[0042] This disclosure provides a method, apparatus, and system for constraining the timing model of haptic effects for ISOBMFF file format tracks containing haptic effects. Two timing options are provided, one of which anchors the media track presentation start time and constrains all haptic effect positions to that time. The other timing option involves each haptic effect being anchored by the sample start time of the effect, and the effect is expressed in sample time. Embodiments include tracks with option 1, tracks with option 2, or mixed tracks.

[0043] Embodiments of the present disclosure provide a timing model that allows haptic effects to be synchronized with other media tracks in the same or related ISOBMFF files. Because the timing model of a haptic track is related to the timing model of the related ISOBMFF file, manipulation and processing of the media track becomes more efficient.

[0044] As shown in FIG. 6, process 600 illustrates an exemplary process for decoding haptic data.

[0045] In operation 605, a media stream including one or more haptic tracks and one or more video tracks can be received.

[0046] In operation 610, one or more Moving Picture Experts Group (MPEG) Immersive Haptic Stream (MIHS) units may be obtained from the media stream. In some embodiments, the MIHS units may include one or more haptic effects. The MIHS units may include a start time for the MIHS units.

[0047] In an embodiment, the MIHS unit is associated with at least one haptic channel, the at least one haptic channel including one or more haptic bands, each of the one or more haptic bands having at least one haptic effect.

[0048] Timing information associated with one or more haptic effects may be obtained in act 615. In embodiments, the timing information may include at least one temporal position of the one or more haptic effects.

[0049] In embodiments, the temporal position of the haptic effect indicates an effect start time for the haptic effect, which is an offset based on the start time of the corresponding MIHS unit. The effect start time may indicate the start time of the haptic effect relative to the start time of the corresponding MIHS unit.

[0050] In an embodiment, the effect start time for a haptic effect is an absolute time based on the start time of at least one haptic track or at least one video track.

[0051] Operation 620 renders the media stream based on the obtained timing information.

[0052] According to an embodiment, manipulating the order of one or more MIHS units does not affect the temporal position of at least one of the one or more haptic effects because the one or more MIHS units correspond to one or more ISO Base Media File Format (ISOBMFF) samples associated with at least one video track.

[0053] In some embodiments, a synchronous MIHS unit can be obtained from a media stream. In embodiments, a synchronous MIHS unit is a special type of MIHS unit that provides a reset point in a bitstream. In embodiments, a synchronous MIHS unit is mapped to a synchronization sample in a video bitstream that corresponds to one or more haptic channels.

[0054] As will be appreciated by those skilled in the art, the techniques described herein can be implemented on both the encoder and decoder sides. The techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, FIG. 7 illustrates a computer system 700 suitable for implementing some embodiments of the present disclosure.

[0055] Computer software can be encoded using any suitable machine code or computer language, which can be assembled, compiled, linked, or similar mechanisms to create code containing instructions that are executed directly by a computer Central Processing Unit (CPU), Graphics Processing Unit (GPU), etc., or interpreted and executed via microcode.

[0056] The instructions may be executed by various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0057] 7 are exemplary and are not intended to suggest any limitation on the scope or functionality of use of computer software for implementing embodiments of the present disclosure. Neither the arrangement of components should be construed as having any dependency or requirement regarding any one or combination of components shown in the non-limiting embodiment of computer system 700.

[0058] The computer system 700 may also include several human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users, for example, through tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, tapping), visual input (e.g., gestures), and olfactory input (not shown). Human-machine interface devices may also be used to capture some media not necessarily directly related to conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, including stereoscopic video, three-dimensional video).

[0059] The input human-machine interface devices may include one or more of a keyboard 701, a mouse 702, a touchpad 703, a touchscreen 710, a data glove, a joystick 705, a microphone 706, a scanner 707, and an imaging device 708 (each illustrating only one).

[0060] The computer system 700 may also include several human-machine interface output devices. Such human-machine interface output devices can stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. These human-machine interface output devices can include haptic output devices (e.g., haptic feedback via a touchscreen 710, data gloves, or joystick 705, although haptic feedback devices not used as input devices may also be present). For example, such devices can include audio output devices (e.g., speakers 709, head-mounted headphones (not shown)), visual output devices (e.g., screens 710, including CRT screens, LCD screens, plasma screens, OLED screens, etc., each with or without touchscreen input capability and each with or without haptic feedback capability, some of which can output two-dimensional visual output or three-dimensional or higher-dimensional output, e.g., in a stereoscopic output format, such as virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0061] The computer system 700 may further include human-accessible storage devices and their associated media, such as optical media including a CD / DVD ROM / RW 720 having media 721 such as a CD / DVD, a thumb drive 722, a removable hard drive or solid state drive 723, traditional magnetic media (not shown) such as magnetic tape or floppy disks, dedicated ROM / ASIC / PLD based devices such as a security dongle (not shown), and the like.

[0062] Those skilled in the art should understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transitory signals.

[0063] The computer system 700 may include interfaces to one or more communication networks. The networks may be, for example, wireless, wired, or optical networks. The networks may be local, wide-area, metropolitan, vehicular, industrial, real-time, delay-tolerant networks, etc. Examples of networks include local networks such as Ethernet, wireless Local Area Networks (LANs), cellular networks including Global System for Mobile Communications (GSM), 3G (the Third Generation, 3G), 4G (the Fourth Generation, 4G), 5G (the Fifth Generation, 5G), and LTE (Long Term Evolution, 5G), TV wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial TV, vehicular and industrial networks including CANBus, etc. Some networks typically require an external network interface adapter connected to some general-purpose data port or peripheral bus 749 (e.g., a USB port on computer system 700), while other networks are typically integrated into the core of computer system 70 by connecting to the system bus via one of the following interfaces (e.g., an Ethernet interface to a personal computer (PC) computer system or a cellular network interface to a smartphone computer system). Using any of these networks, computer system 700 can communicate with other entities. Such communications may be one-way receive only (e.g., broadcast TV), one-way transmit only (e.g., a Controller Area Network (CAN) bus to some CAN bus devices), or bidirectional, e.g., via a local-area or wide-area digital network to another computer system. Such communications may include communications to a cloud computing environment 755.Certain protocols and protocol stacks may be used in each of these networks and network interfaces described above.

[0064] The aforementioned human machine interface devices, human accessible storage devices and network interface 754 may be attached to the core 740 of the computer system 700 .

[0065] The core 740 may include one or more central processing units (CPUs) 741, graphics processing units (GPUs) 742, dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) 743, hardware accelerators 744 for some specific tasks, etc. These devices, along with read-only memory (ROM) 745, random access memory 746, and internal mass storage device 747, such as an internal hard disk drive or SSD that is not user accessible, are connected via a system bus 748. In some computer systems, access to the system bus 748 in the form of one or more physical plugs may allow expansion with additional CPUs, GPUs, etc. Peripheral devices are connected to the core's system bus 748 directly or via a peripheral bus 749. Peripheral bus architectures include Peripheral Component Interconnect / Interface (PCI), Universal Serial Bus (USB), etc. A graphics adapter 750 may also be included in the core 740.

[0066] The CPU 741, GPU 742, FPGA 743, and accelerator 744 may execute certain instructions, which may combine to constitute the computer code described above. The computer code may be stored in a ROM 745 or a random access memory (RAM) 746. Temporary data may also be stored in the RAM 746, while persistent data may be stored, for example, in an internal mass storage device 747. A cache memory may be used to achieve high-speed storage and retrieval from any memory device, and the cache memory may be closely associated with one or more of the CPU 741, GPU 742, mass storage device 747, ROM 745, RAM 746, etc.

[0067] The computer-readable medium may have computer code thereon for performing various computer-implemented operations, and the medium and computer code may be specially designed and constructed for the purposes of this disclosure, or may be of the type known and available to those skilled in the computer software arts.

[0068] By way of example and not limitation, a computer system having the architecture of computer system 700, and in particular core 740, can provide functionality through the execution of software embodied in one or more tangible computer-readable media by a processor (including a CPU, GPU, FPGA, accelerator, etc.). Such computer-readable media may be media associated with user-accessible mass storage devices, as described above, and some storage devices of a non-transitory nature (e.g., core internal mass storage device 747 and ROM 745) of core 740. Software of various embodiments for implementing the present disclosure can be stored on such devices and executed by the core. Depending on particular needs, the computer-readable media may include one or more storage devices or chips. The software can cause the core, and in particular the processor therein (including a CPU, GPU, FPGA, etc.), to perform particular processes or portions of particular processes, as described herein, including defining data structures stored in RAM 746 and modifying such data structures according to software-defined processes. Additionally, or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 744), which may operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry (e.g., integrated circuits (ICs)) that store software for execution, circuitry that implements logic for execution, or both. The present disclosure includes any appropriate combination of hardware and software.

[0069] While this disclosure has previously described several non-limiting embodiments, there are alterations, substitutions, and various substitute equivalents that fall within the scope of this disclosure. Thus, those skilled in the art may devise various systems and methods that, although not explicitly shown or described herein, embody the principles of this disclosure and are therefore within the spirit and scope of this disclosure.

Claims

1. 1. A method for decoding timing information of haptic data, executed by at least one processor, comprising: The method comprises: receiving a media stream including at least one haptic track and at least one video track; receiving a Motion Picture Experts Group (MPEG) Immersive Haptic Stream (MIHS) unit from the media stream, the MIHS unit including one or more haptic effects and a start time; obtaining timing information associated with the one or more haptic effects from the media stream, the timing information including at least one temporal position of the one or more haptic effects; rendering the media stream based on the obtained timing information; A method comprising:

2. the temporal position of the haptic effect indicates an effect start time for the haptic effect; The method of claim 1.

3. an effect start time for the haptic effect that is an offset based on the start time of the corresponding MIHS unit; the effect start time indicates a start time of the haptic effect relative to a start time of the corresponding MIHS unit. The method of claim 2.

4. Manipulating the order of the MIHS units does not affect the temporal location of the at least one of the one or more haptic effects. The method of claim 1.

5. the MIHS unit is associated with at least one haptic channel; the at least one tactile channel includes one or more tactile bands; Each of the one or more haptic bands has at least one haptic effect. The method of claim 1.

6. the MIHS units correspond to ISO Base Media File Format (ISOBMFF) samples associated with the at least one video track; The method of claim 1.

7. an effect start time for the haptic effect is an absolute time based on a start time of the at least one haptic track or the at least one video track; The method of claim 2.

8. 1. An apparatus for decoding haptic data, the apparatus comprising: at least one memory configured to store program code; at least one processor configured to read said program code and to act in accordance with the instructions of said program code; Including, The program code first receiving code arranged to cause the at least one processor to receive a media stream including at least one haptic track and at least one video track; first acquisition code configured to cause the at least one processor to acquire a Motion Picture Experts Group (MPEG) Immersive Haptic Stream (MIHS) unit from the media stream, the MIHS unit including one or more haptic effects and a start time; second acquisition code arranged to cause the at least one processor to acquire, from the media stream, timing information associated with the one or more haptic effects, the timing information including at least one temporal position of the one or more haptic effects; and rendering code configured to cause the at least one processor to render the media stream based on the obtained timing information; An apparatus comprising:

9. the temporal position of the haptic effect indicates an effect start time for the haptic effect; 9. The apparatus of claim 8.

10. an effect start time for the haptic effect that is an offset based on the start time of the corresponding MIHS unit; the effect start time indicates a start time of the haptic effect relative to a start time of the corresponding MIHS unit.

10. The apparatus of claim 9.

11. Manipulating the order of the MIHS units does not affect the temporal location of the at least one of the one or more haptic effects.

9. The apparatus of claim 8.

12. the MIHS unit is associated with at least one haptic channel; the at least one tactile channel includes one or more tactile bands; Each of the one or more haptic bands has at least one haptic effect.

9. The apparatus of claim 8.

13. the MIHS units correspond to ISO Base Media File Format (ISOBMFF) samples associated with the at least one video track; 9. The apparatus of claim 8.

14. an effect start time for the haptic effect is an absolute time based on a start time of the at least one haptic track or the at least one video track; 10. The apparatus of claim 9.

15. A computer-executable program containing instructions, The instructions include one or more instructions that, when executed by one or more processors of an apparatus for decoding haptic data, cause the one or more processors to: A program for executing the method according to any one of claims 1 to 7.

Images